A pair of torch.utils.data.Datasets. Args: src (torch.utils.data.Dataset): source dataset to wrap src_sizes (List[int]): sou

Break a Dataset of tokens into blocks. Args: dataset (~torch.utils.data.Dataset): dataset to break into blocks sizes (List[int]):

Multi-headed attention. See "Attention Is All You Need" for more details.

A mapping from symbols to consecutive integers

Records progress through the set of constraints for each item in the beam using a trie.

This is an implementation of the efficient softmax approximation for graphical processing units (GPU), described in the paper "Efficient soft

Records progress through the set of linear nonbranching constraints with gaps.

1D convolution over an input of shape (time x batch x channel) The implementation uses gemm to perform the convolution. This implementation i

A sorted dictionary of :class:`Meters`. Meters are sorted according to a priority that is given when the meter is first added to the dictiona

Wrap a DistributedDataParallel module and forward requests for missing attributes to the module wrapped by DDP (the twice-wrapped module).

A simple PyTorch Hub interface to RoBERTa. Usage: https://github.com/pytorch/fairseq/tree/master/examples/roberta

Truncate a sequence by returning the first truncation_length tokens

A LayerDrop implementation based on :class:`torch.nn.ModuleList`. We refresh the choice of which layers to drop every time we iterate ov

This module learns positional embeddings up to a fixed maximum size. Padding ids are ignored by either offsetting based on padding_idx or

Model parallel Multi-headed attention. This performs the Multi-headed attention over multiple gpus. See "Megatron-LM: https://arxiv.org/pdf/1

Randomly samples from a given dataset at each epoch. Sampling is done with or without replacement, depending on the "replace" parameter.

Zip multiple :class:`~fairseq.data.FairseqDataset` instances together. Shorter datasets are repeated in a round-robin fashion to match the length

Scores the target for a given source sentence.

Computes the sum/avg duration of some event in seconds

Encoder layer block. In the original paper each operation (multi-head attention or FFN) is postprocessed with: `dropout -> add residual -> la

r"""Implements Adam algorithm. This implementation is modified from torch.optim.Adam based on: `Fixed Weight Decay Regularization in Adam`

Bucket and pad item lengths to the nearest bucket size. This can be used to reduce the number of unique batch shapes, which is important on T

Represents a node in a trie managing unordered constraints.

Wrapper around an iterable that maintains the iteration count. Args: iterable (iterable): iterable to wrap start (int): starting

A wrapper around TokenBlockDataset for BART dataset. Args: dataset (TokenBlockDataset): dataset to wrap sizes (List[int]): s

Selects every nth element, where n is the index

Multi-headed attention with Gating and Downsampling

A generic evaluation-time tokenizer, which leverages built-in tokenizers in sacreBLEU (https://github.com/mjpost/sacrebleu). It additionally provi

Convolutional decoder

Loader for TorchNet IndexedDataset

LSTM decoder.

LightConv decoder consisting of *args.decoder_layers* layers. Each layer is a :class:`LightConvDecoderLayer`. Args: args (argpar

An optimized version of nn.Conv1d. At training time, this module uses ConvTBC, which is an optimized version of Conv1d. At inference time, it

A wrapper Dataset for masked language modelling. The dataset wraps around TokenBlockDataset or BlockedPairDataset and creates a batch whe

Model Parallel Transformer decoder consisting of *args.decoder_layers* layers. Each layer is a :class:`ModelParallelTransformerDecoderLayer`.

Model parallel Transformer encoder consisting of *args.encoder_layers* layers. Each layer is a :class:`ModelParallelTransformerEncoderLayer`.

Source: https://stackoverflow.com/a/54128391

Head for masked language modeling.

Wrapper class for data config YAML

Samples from multiple sub-datasets according to given sampling ratios. Args: datasets ( List[~torch.utils.data.Dataset]

A sharded wrapper around an iterable, padded to length. Args: iterable (iterable): iterable to wrap num_shards (int): number of s

Single-head attention that supports Gating and Downsampling

Computes the average occurrence of some event per second

A :class:`~fairseq.data.FairseqDataset` wrapper that transform bos on collated samples of language pair dataset. Note that the transformation

Transformer decoder consisting of *args.decoder_layers* layers. Each layer is a :class:`TransformerDecoderLayer`. Args: args (ar

Transformer decoder consisting of *args.decoder_layers* layers. Each layer is a :class:`TransformerDecoderLayer`. Args: args (ar

Decoder layer block. In the original paper each operation (multi-head attention, encoder attention or FFN) is postprocessed with: `dropout ->

Transformer encoder consisting of *args.encoder_layers* layers. Each layer is a :class:`TransformerEncoderLayer`. Args: args (ar

Transformer encoder consisting of *args.encoder_layers* layers. Each layer is a :class:`TransformerEncoderLayer`. Args: args (ar

Fake scalar quantization of the activations using a forward hook. Args: - module. a nn.Module for which we quantize the *post-activa

Implements Adafactor algorithm. This implementation is based on: `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost` (see htt

Implements Adamax algorithm (a variant of Adam based on infinity norm). It has been proposed in `Adam: A Method for Stochastic Optimization`__.

Computes and stores the average and current value

Log to Azure ML

Head for sentence-level classification tasks.