BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: config: a BertConfig class instance with the configura

A single set of features of data.

Runs basic tokenization (punctuation splitting, lower casing, etc.).

Implements BERT version of Adam algorithm with weight decay fix. Params: lr: learning rate warmup: portion of t_total for the warm

Configuration class to store the configuration of a `BertModel`.

Construct the embeddings from word, position and token_type embeddings.

BERT model with pre-training heads. This module comprises the BERT model followed by the two pre-training heads: - the masked language mod

Runs WordPiece tokenization.

BERT model with the masked language modeling head. This module comprises the BERT model followed by the masked language modeling head. Params

BERT model for multiple choice tasks. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model with next sentence prediction head. This module comprises the BERT model followed by the next sentence classification head. Params

BERT model for Question Answering (span extraction). This module is composed of the BERT model with a linear layer on top of the sequence outp

BERT model for classification. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model for token-level classification. This module is composed of the BERT model with a linear layer on top of the full hidden state of th

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

Runs end-to-end tokenization: punctuation splitting + wordpiece

BERT model with pre-training heads. This module comprises the BERT model followed by the two pre-training heads: - the masked language mod