hiddens: list (num_layers) of [batch, query_length, 2d] mems: None or [num_layers, batch, memory_length, 2d]

Open URLs and yield a stream of url+stream pairs. Args: data: iterator over dict(url=...) handler: exception handler. kw:

Zero out the parameters of a module and return it.

Zero out the parameters of a module and return it.

convert Id to sentencpiece token

convert Id to token accounting for command tokens

convert token to Id accounting for command tokens

extracts samples only pertaining to this worker's batch

Build the size on rank 0 and broadcast.

Check that all the keys have the same target data type.

Performs invalid character removal and whitespace cleanup on text.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transf

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Necessary initialization for torch.distributed for model-only mode

Adds whitespace around any CJK character.

r"""Copy parameters and buffers from :attr:`state_dict` into only this module, but not its descendants. This is called on every submodule in

Computes the precision@k for the specified values of k

Evaluation arguments.

Evaluation arguments.

Model arguments

Sampling configurations

Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current vocabula

Text generate arguments.

tokenization arguments.

Perform random_masking after adding position_embedding.