append an existing dataset. Beware, if it wasn't built with the same coder, you are in trouble.

Returns obj[k] if it exists and is not None, otherwise returns default.

Helper to get index of pad symbol

Helper to get index of end-of-sentence symbol

Fetch a read only view of an np.array, stored in plasma.

Loads the dictionary from a text file with the format: ``` <symbol0> <count0> <symbol1> <count1> ... ```

Helper to get index of beginning-of-sentence symbol

Log intermediate stats according to log_interval.

return the Enum class used to enforce list of choices

Adds a word to the dictionary

Helper to get index of unk symbol

Returns the index of the specified symbol

Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described

Returns True if the given key exists and is not None.

Helper for setting incremental state for an nn.Module.

Helper for getting incremental state for an nn.Module.

convert a dataclass instance to tailing parser arguments. If `with_prefix` is provided, prefix all the keys in the resulting parser with it.

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Stores dictionary into a text file

Weight-normalized Linear layer (input: N x T x C)

Filling tensor x with y at masked positions (dim=0).

Helper for converting a tensor of token indices to a string. Can optionally remove BPE symbols or escape <unk> words.

Get the number of parameters updates.

Updates counts from new dictionary.

Build the pre-tokenizer for this task.

Computes random mask spans for a given shape Args: shape: the the shape for which to compute masks. should be of size 2

Convert a flat argparse.Namespace to a structured DictConfig.

Weight-normalized Linear layer (input: B x T x C)

Weight-normalized Linear layer (input: N x T x C)

Build a :class:`~fairseq.SequenceGenerator` instance for this task. Args: models (List[~fairseq.models.FairseqMo

Get the (non-wrapped) model instance.

Maximum length supported by the model.