Cut a sentence based on the numbers of tokens to be cut from beginning and end Represent the sentence as dataset idx and return

Fetch a block of tokens based on its dataset idx

Return language ID index.

Aggregate training time in seconds.

Mask tokens for Masked Language Model training Samples mask_ratio tokens that will be predicted by LM. Note:This function ma

Generate a random integer which is not in skip_ids. Sample range is [0, total) TODO: ids in skip_ids should be consecutive, we can ex

In order to be efficient, the AdaptiveSoftMax does not compute the scores for all the word of the vocabulary for all the examples. It

increments counters for the sum of log probs of current word and next word (given context ending at current word). Since the next word might b

Given an ordered set of indices, return batches according to *max_tokens*, *max_sentences* and *required_batch_size_multiple*.

left and right trim

Broadcasts a list of tensors without other (non-src) ranks needing to know the dtypes/shapes of the tensors.

A friendlier wrapper for performing activation checkpointing. Compared to the PyTorch version, this version: - wraps an nn.Module, so th

Helper to get index of cls symbol

Function to compute the cross entropy loss. The default value of ignore_index is the same as the default value for F.cross_entropy in pyt

Computes random mask spans for a given shape Args: shape: the the shape for which to compute masks. should be of size 2

Converts a yaml dictionary with two keys: `key` and `value` into a two argument tuple of those values.

Performs self.n_iter EM steps.

BPE-encode a sentence (or multiple sentences). Every sequence begins with a beginning-of-sentence (`<s>`) symbol. Every sent

Extends convolutional spec that is a list of tuples of 2 or 3 parameters (kernel size, dim size and optionally how many layers behind to look

Similar to *forward* but only return features.