Helper to get index of end-of-sentence symbol

Helper to get index of pad symbol

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Adds a word to the dictionary

Log intermediate stats according to log_interval.

Loads the dictionary from a text file with the format: ``` <symbol0> <count0> <symbol1> <count1> ... ```

Helper to get index of unk symbol

Returns the index of the specified symbol

Helper to get index of beginning-of-sentence symbol

Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described

Stores dictionary into a text file

Helper for converting a tensor of token indices to a string. Can optionally remove BPE symbols or escape <unk> words.

Get the number of parameters updates.

Get all smoothed values.

Helper for setting incremental state for an nn.Module.

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Helper for getting incremental state for an nn.Module.

Weight-normalized Linear layer (input: N x T x C)

Filling tensor x with y at masked positions (dim=0).

Add optimizer-specific arguments to the parser.

This function must be called explicitly after backward to reduce gradients. There is no automatic hook like c10d.

Maximum input length supported by the encoder.

Return a new iterator over the dataset. Args: shuffle (bool, optional): shuffle batches before returning the iter

Generate a batch of translations. Args: sample (dict): batch prefix_tokens (torch.LongTensor, optional): force decode

Whether the iterator has been exhausted.

Build the :class:`~fairseq.criterions.FairseqCriterion` instance for this task. Args: args (argparse.Namespace):

Weight-normalized Linear layer (input: B x T x C)

Weight-normalized Linear layer (input: N x T x C)

Return a loaded dataset split. Args: split (str): name of the split (e.g., train, valid, test) Returns:

Sort symbols by frequency in descending order, ignoring special ones. Args: - threshold defines the minimum word count

Maximum length supported by the model.

register dataclasses defined in modules in config store, for example, in migrated tasks, models, etc.

Take a single search step. Args: step: the current search step, starting at 0 lprobs: (bsz x input_beam_size x vocab_

return the Enum class used to enforce list of choices

Args: append_eos: if True, each input sentence in the source tokens tensor will have an EOS appended to the end.

This verifies that with a given x, x_len, max_shuffle_distance, and vocab, we get the expected shuffle result. Args:

convert a dataclass instance to tailing parser arguments

Get an iterator that yields batches of data from the given dataset. Args: dataset (~fairseq.data.FairseqDataset): datase

Load all training state from a checkpoint file.

Truncates the iterator to n elements at most.

Linear layer (input: N x T x C)