The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added betwee

Roberta Head for masked language modeling.

For parallel adapter.

Head for sentence-level classification tasks.

Same as BertEmbeddings with a tiny tweak for positional embeddings indexing.

Implements BERT version of Adam algorithm with weight decay fix. Params: lr: learning rate warmup: portion of t_total for the warm

An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models.