Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Helper to get index of pad symbol

Helper to get index of end-of-sentence symbol

Loads the dictionary from a text file with the format: ``` <symbol0> <count0> <symbol1> <count1> ... ```

Adds a word to the dictionary

Fetch a read only view of an np.array, stored in plasma.

Log intermediate stats according to log_interval.

Returns the index of the specified symbol

Helper to get index of unk symbol

Helper to get index of beginning-of-sentence symbol

Helper for converting a tensor of token indices to a string. Can optionally remove BPE symbols or escape <unk> words.

return the Enum class used to enforce list of choices

Convert a list of 1d tensors into a padded 2d tensor.

Convert a flat argparse.Namespace to a structured DictConfig.

Stores dictionary into a text file

BufferGetAndValidate */

Helper for setting incremental state for an nn.Module.

BufferGetAndValidate */

Helper for getting incremental state for an nn.Module.

Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described

Get the number of parameters updates.

Get all smoothed values.

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Add optimizer-specific arguments to the parser.

convert a dataclass instance to tailing parser arguments

Weight-normalized Linear layer (input: N x T x C)

Filling tensor x with y at masked positions (dim=0).

CIntToPy */

ExtTypeTest */

Maximum input length supported by the encoder.

Build the tokenizer for this task.

Whether the iterator has been exhausted.