BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: config: a BertConfig class instance with the configura

Configuration class to store the configuration of a `GPT2Model`.

OpenAI GPT-2 model ("Language Models are Unsupervised Multitask Learners"). Params: config: a GPT2Config class instance with the configur

Configuration class to store the configuration of a `OpenAIGPTModel`.

OpenAI GPT model ("Improving Language Understanding by Generative Pre-Training"). OpenAI GPT use a single embedding matrix to store the word and

Configuration class to store the configuration of a `TransfoXLModel`.

Runs basic tokenization (punctuation splitting, lower casing, etc.).

Language Model Head for the transformer

Language Model Head for the transformer

Implements BERT version of Adam algorithm with weight decay fix. Params: lr: learning rate warmup: portion of t_total for the warm

Configuration class to store the configuration of a `BertModel`.

Construct the embeddings from word, position and token_type embeddings.

BERT model with pre-training heads. This module comprises the BERT model followed by the two pre-training heads: - the masked language mod

Classifier Head for the transformer

Classifier Head for the transformer

Transformer XL model ("Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context"). This model add an (adaptive) softmax head on to

Transformer XL model ("Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context"). Transformer XL use a relative positioning (with

Transformer-XL tokenizer adapted from Vocab class in https://github.com/kimiyoung/transformer-xl

Runs WordPiece tokenization.

BERT model with the masked language modeling head. This module comprises the BERT model followed by the masked language modeling head. Params

BERT model for multiple choice tasks. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model with next sentence prediction head. This module comprises the BERT model followed by the next sentence classification head. Params

BERT model for Question Answering (span extraction). This module is composed of the BERT model with a linear layer on top of the sequence outp

BERT model for classification. This module is composed of the BERT model with a linear layer on top of the pooled output. Params:

BERT model for token-level classification. This module is composed of the BERT model with a linear layer on top of the full hidden state of th

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

Runs end-to-end tokenization: punctuation splitting + wordpiece

配置参数

配置参数

配置参数

配置参数

配置参数

配置参数

OpenAI GPT-2 model with a Language Modeling and a Multiple Choice head ("Language Models are Unsupervised Multitask Learners"). Params: c

OpenAI GPT-2 model with a Language Modeling head ("Language Models are Unsupervised Multitask Learners"). Params: config: a GPT2Config cl

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

GPT-2 BPE tokenizer. Peculiarities: - Byte-level BPE

Implements Open AI version of Adam algorithm with weight decay fix.

OpenAI GPT model with a Language Modeling and a Multiple Choice head ("Improving Language Understanding by Generative Pre-Training"). OpenAI GPT

OpenAI GPT model with a Language Modeling head ("Improving Language Understanding by Generative Pre-Training"). OpenAI GPT use a single embedding

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

BPE tokenizer. Peculiarities: - lower case all inputs - uses SpaCy tokenizer and ftfy for pre-BPE tokenization if they are instal

An abstract class to handle weights initialization and a simple interface for dowloading and loading pretrained models.

Linearly increases learning rate from 0 to 1 over `warmup` fraction of training steps. Keeps learning rate equal to 1. after warmup.

Linearly increases learning rate from 0 to 1 over `warmup` fraction of training steps. Decreases learning rate from 1. to 0. over remaining `

Linearly increases learning rate from 0 to 1 over `warmup` fraction of training steps. If `cycles` (default=1.) is different from default, le

All training progress is divided in `cycles` (default=1.) parts of equal length. Every part follows a schedule with the first `warmup` fracti

Linearly increases learning rate from 0 to 1 over `warmup` fraction of training steps. Linearly decreases learning rate from 1. to 0. over re

Parent of all LRSchedules here.