BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: config: a BertConfig class instance with the configura

Runs basic tokenization (punctuation splitting, lower casing, etc.).

Implements BERT version of Adam algorithm with weight decay fix. Params: lr: learning rate warmup: portion of t_total for the warm

Configuration class to store the configuration of a `BertModel`.

Construct the embeddings from word, position and token_type embeddings.

BERT model with the masked language modeling head. This module comprises the BERT model followed by the masked language modeling head. Params

Runs end-to-end tokenization: punctuation splitting + wordpiece

A single set of features of data.

Runs WordPiece tokenization.

BERT model for multiple choice tasks. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model with next sentence prediction head. This module comprises the BERT model followed by the next sentence classification head. Params

BERT model with pre-training heads. This module comprises the BERT model followed by the two pre-training heads: - the masked language mod

BERT model for Question Answering (span extraction). This module is composed of the BERT model with a linear layer on top of the sequence outp

BERT model for classification. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model for token-level classification. This module is composed of the BERT model with a linear layer on top of the full hidden state of th

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

Applies linear warmup. After warmup always returns 1..

Cosine learning rate schedule with linear warmup. Cosine after warmup is without restarts.

Cosine learning rate schedule with linear warmup and hard restarts (if cycles > 1).

Cosine learning rate schedule with linear warmups and linear warmup restarts. The same warmup rate is used for warmup restarts as for initial

Linear warmup. Linear decay after warmup.

Parent of all LRSchedules here.