BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: config: a BertConfig class instance with the configura

Implements BERT version of Adam algorithm with weight decay fix. Params: lr: learning rate warmup: portion of t_total for the w

Track a series of values and provide access to smoothed values over a window or the global series average.

Track a series of values and provide access to smoothed values over a window or the global series average.

Runs basic tokenization (punctuation splitting, lower casing, etc.).

Configuration class to store the configuration of a `BertModel`.

Construct the embeddings from word, position and token_type embeddings.

Apply bpe tokenization to str line

Multimodal BERT : Masked Language Model + Image Text Matching

2-class classification model : Aligned, Not aligned

2-class classification model : Aligned, Not aligned

With label smoothing, KL-divergence between q_{smoothed ground truth prob.}(w) and p_{prob. computed by model}(w) is minimized.

Multimodal BERT : Masked Language Model + Image Text Matching

Pre-processing steps for pretraining transformer

Runs WordPiece tokenization.

refer to BertForPreTraining

refer to BertForPreTraining

Runs end-to-end tokenization: punctuation splitting + wordpiece

Apply bpe online to data in parallel

Gather tensors from all process, supporting backward propagation.

Load image-sentence pairs

(vocab_size) classification model

Pre-process Pipeline Class : callable

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

Pre-processing steps for pretraining transformer