hub / github.com/HA-Transformer/MAT / TokenBlockDataset

Class TokenBlockDataset

fairseq/data/token_block_dataset.py:14–194 · view source on GitHub ↗

Break a Dataset of tokens into blocks. Args: dataset (~torch.utils.data.Dataset): dataset to break into blocks sizes (List[int]): sentence lengths (required for 'complete' and 'eos') block_size (int): maximum block size (ignored in 'eos' break mode) break_mode (s

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 8

load_datasetMethod · 0.90

build_dataset_for_inferenceMethod · 0.90

load_datasetMethod · 0.90

build_dataset_for_inferenceMethod · 0.90

_load_single_lang_datasetMethod · 0.90

setUpMethod · 0.90

_build_datasetMethod · 0.90

Calls

no outgoing calls

Tested by 3

setUpMethod · 0.72

_build_datasetMethod · 0.72