The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added betwee

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added betwee

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added betwee

A wrapper around a simple schedule function.

A wrapper around a simple schedule function.

Roberta Head for masked language modeling.

Construct the embeddings from word, position and token_type embeddings.

Construct the embeddings from word, position and token_type embeddings.

Output type of [`BertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,

Output type of [`BertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,

Head for sentence-level classification tasks.

Same as BertEmbeddings with a tiny tweak for positional embeddings indexing.

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.

Base class for all matrix-noise schedulers.

Base class for all matrix-noise schedulers.

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.