Helper to get index of pad symbol

Helper to get index of end-of-sentence symbol

Adds a word to the dictionary

Get a specific meter by name.

Log intermediate stats according to log_interval.

Helper to get index of unk symbol

Loads the dictionary from a text file with the format: ``` <symbol0> <count0> <symbol1> <count1> ... ```

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Maximum input length supported by the encoder.

Get the number of parameters updates.

Weight-normalized Linear layer (input: N x T x C)

Return a loaded dataset split. Args: split (str): name of the split (e.g., train, valid, test) Returns:

Return a new iterator over the dataset. Args: shuffle (bool, optional): shuffle batches before returning the iter

Helper for converting a tensor of token indices to a string. Can optionally remove BPE symbols or escape <unk> words.

Updates counts from new dictionary.

Get an iterator that yields batches of data from the given dataset. Args: dataset (~fairseq.data.FairseqDataset): datase

Merge branches. branch_outputs: list of Tensor or tuple of Tensors

Stores dictionary into a text file

Weight-normalized Linear layer (input: B x T x C)

Weight-normalized Linear layer (input: N x T x C)

Sort symbols by frequency in descending order, ignoring special ones. Args: - threshold defines the minimum word count

Generate a batch of translations. Args: models (List[~fairseq.models.FairseqModel]): ensemble of models sample (dict)

Maximum length supported by the model.

Args: append_eos: if True, each input sentence in the source tokens tensor will have an EOS appended to the end.

Add optimizer-specific arguments to the parser.

This verifies that with a given x, x_len, max_shuffle_distance, and vocab, we get the expected shuffle result. Args:

Load a given dataset split. Args: split (str): name of the split (e.g., train, valid, test)

Helper to get index of beginning-of-sentence symbol

Score a batch of translations.

Whether the iterator has been exhausted.

Linear layer (input: N x T x C)

Linear layer (input: N x T x C)

increments counters for the sum of log probs of current word and next word (given context ending at current word). Since the next word mi

Returns the index of the specified symbol

Load all training state from a checkpoint file.

Load the dictionary from the filename Args: filename (str): the filename

Print end-of-epoch stats.