If dataset A has length N and dataset B has length M then index 1 maps to index 1 of dataset A, and index N + 1 maps to index

Helper to set default arguments based on *add_args*.

Build a new model instance.

Retrieves all checkpoints found in `path` directory. Checkpoints are identified by matching filename to the specified pattern. If the pattern

Retrieves all checkpoints found in `path` directory. Checkpoints are identified by matching filename to the specified pattern. If the pattern

given a file and a number of chuncks, find the offsets in the file to be able to chunk around full lines.

Generate a batch of inferences.

A helper function for loading indexed datasets. Args: path (str): path to indexed dataset (e.g., 'data-bin/train') dictionary (~f

Maximum output length supported by the encoder.

Return a Tensor of `size` filled with a range function on the device of x. If size is empty, using the size of the variable x.

Load data from a model spec into EMA model

Returns a counter of the number of times each token is used in a constraint.

Log latest configuration.

Convert a list of 2D frames into a padded 3D tensor Args: frames (list): list of 2D frames of size L[i]*f_dim. Where L[i] is

Return language token index.

Different underlying datasets have different lengths. In order to ensure we are not accessing an index outside the range of the curre

Loads a pre-existing dictionary from a text file and adds its symbols to this instance.

Reads in a token and advances the state. Here's how it works. We can advance to the next state if: - there is a matching child