Helper to get index of pad symbol

Helper to get index of end-of-sentence symbol

Log intermediate stats according to log_interval.

Helper to get index of beginning-of-sentence symbol

Adds a word to the dictionary

Helper to get index of unk symbol

Loads the dictionary from a text file with the format: ``` <symbol0> <count0> <symbol1> <count1> ... ```

return the Enum class used to enforce list of choices

Get the number of parameters updates.

Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described

Returns the index of the specified symbol

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Convert a flat argparse.Namespace to a structured DictConfig.

convert a dataclass instance to tailing parser arguments

Weight-normalized Linear layer (input: N x T x C)

Helper for converting a tensor of token indices to a string. Can optionally remove BPE symbols or escape <unk> words.

Helper for setting incremental state for an nn.Module.

Helper for getting incremental state for an nn.Module.

Maximum input length supported by the encoder.

Weight-normalized Linear layer (input: B x T x C)

Weight-normalized Linear layer (input: N x T x C)

Maximum length supported by the model.

Pad Dictionary size to be a multiple of *padding_factor*.

Build the tokenizer for this task.

Return a loaded dataset split. Args: split (str): name of the split (e.g., train, valid, test) Returns:

Sort symbols by frequency in descending order, ignoring special ones. Args: - threshold defines the minimum word count

Get an iterator that yields batches of data from the given dataset. Args: dataset (~fairseq.data.FairseqDataset): datase

Get the (non-wrapped) criterion instance.

Get all smoothed values.

Load the dictionary from the filename Args: filename (str): the filename

Return a new iterator over the dataset. Args: shuffle (bool, optional): shuffle batches before returning the iter

Stores dictionary into a text file

Updates counts from new dictionary.

Linear layer (input: N x T x C)

Build the pre-tokenizer for this task.

Print end-of-epoch stats.