append an existing dataset. Beware, if it wasn't built with the same coder, you are in trouble.

Helper to get index of pad symbol

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Helper to get index of end-of-sentence symbol

Returns obj[k] if it exists and is not None, otherwise returns default.

Loads the dictionary from a text file with the format: ``` <symbol0> <count0> <symbol1> <count1> ... ```

Fetch a read only view of an np.array, stored in plasma.

Adds a word to the dictionary

Log intermediate stats according to log_interval.

Returns the index of the specified symbol

Helper to get index of beginning-of-sentence symbol

Helper to get index of unk symbol

Helper for converting a tensor of token indices to a string. Can optionally remove BPE symbols or escape <unk> words.

return the Enum class used to enforce list of choices

Stores dictionary into a text file

Convert a flat argparse.Namespace to a structured DictConfig.

Helper for setting incremental state for an nn.Module.

Helper for getting incremental state for an nn.Module.

Wraps modules and applies quantization noise to the weights for subsequent quantization with Iterative Product Quantization as described

convert a dataclass instance to tailing parser arguments. If `with_prefix` is provided, prefix all the keys in the resulting parser

Get all smoothed values.

Returns True if the given key exists and is not None.

Get the number of parameters updates.

Get the number of parameters updates.

Returns if the word is in the trie. :type word: str :rtype: bool

Return an example's size as a float or tuple. This value is used when filtering a dataset with ``--max-positions``.

Add optimizer-specific arguments to the parser.

Build the tokenizer for this task.

Weight-normalized Linear layer (input: N x T x C)

Filling tensor x with y at masked positions (dim=0).

Build the pre-tokenizer for this task.

Take a single search step. Args: step: the current search step, starting at 0 lprobs: (bsz x input_beam_size x vocab_

Get normalized probabilities (or log probs) from a net's output.

Return a new iterator over the dataset. Args: shuffle (bool, optional): shuffle batches before returning the iter