Return a new iterator over the dataset. Args: shuffle (bool, optional): shuffle batches before returning the iter

Generate a batch of translations. Args: sample (dict): batch prefix_tokens (torch.LongTensor, optional): force decode

ExtTypeTest */

Build the :class:`~fairseq.criterions.FairseqCriterion` instance for this task. Args: cfg (omegaconf.DictConfig)

Build the pre-tokenizer for this task.

Evaluate language model based on the current lm state and new word Parameters: ----------- state: current lm state

Take a single search step. Args: step: the current search step, starting at 0 lprobs: (bsz x input_beam_size x vocab_

Weight-normalized Linear layer (input: B x T x C)

Weight-normalized Linear layer (input: N x T x C)

KeywordStringCheck */

Sort symbols by frequency in descending order, ignoring special ones. Args: - threshold defines the minimum word count

Maximum length supported by the model.

Pad Dictionary size to be a multiple of *padding_factor*.

Truncate the iterator to n elements at most.

Updates counts from new dictionary.

KeywordStringCheck */

MemviewSliceIsContig */

MemviewSliceIsContig */

Args: append_eos: if True, each input sentence in the source tokens tensor will have an EOS appended to the end.

This verifies that with a given x, x_len, max_shuffle_distance, and vocab, we get the expected shuffle result. Args:

Return a loaded dataset split. Args: split (str): name of the split (e.g., train, valid, test) Returns:

Get an iterator that yields batches of data from the given dataset. Args: dataset (~fairseq.data.FairseqDataset): datase

Load all training state from a checkpoint file. rank = 0 will load the checkpoint, and then broadcast it to all other ranks.

Load a given dataset split. Args: split (str): name of the split (e.g., train, valid, test)

Load the dictionary from the filename Args: filename (str): the filename

Linear layer (input: N x T x C)

CIntFromPy */

CIntToPy */

PyObjectCallOneArg */

RaiseUnexpectedTypeError */

Args: cfg (omegaconf.DictConfig): fairseq args params (iterable): iterable of parameters to optimize

Computes random mask spans for a given shape Args: shape: the the shape for which to compute masks. should be of size 2

Get the (non-wrapped) criterion instance.

Prepare model for inference.

To leverage the efficiency of distributed training, samples for all layers are computed at once for each logit_idx. Logits are parameters

Linear layer (input: N x T x C)