BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: `config`: a BertConfig class instance with the configu

BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: `config`: a BertConfig class instance with the configu

Configuration class to store the configuration of a `BertModel`.

Configuration class to store the configuration of a `BertModel`.

Construct the embeddings from word, position and token_type embeddings.

Construct the embeddings from word, position and token_type embeddings.

A single set of features of data.

used for simple decoding based on similarity with output embeddings.

Span extraction head for QA. Mostly borrowed from https://github.com/raylin1000/drop-bert/blob/master/drop_bert/augmented_bert.py .

A single training/test example for the Squad dataset. For examples without an answer, the start and end position are -1.

BERT model with the masked language modeling head. This module comprises the BERT model followed by the masked language modeling head. Params

BERT model for multiple choice tasks. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model with next sentence prediction head. This module comprises the BERT model followed by the next sentence classification head. Params

BERT model with pre-training heads. This module comprises the BERT model followed by the two pre-training heads: - the masked language mod

BERT model for Question Answering (span extraction). This module is composed of the BERT model with a linear layer on top of the sequence outp

BERT model for classification. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model for token-level classification. This module is composed of the BERT model with a linear layer on top of the full hidden state of th

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

Bert encoder + Bert decoder with extra mlm, span extraction, etc heads. TODO: for exractive QA, include BIO tagging head.