A raw input example consisting of one or two segments of text and a label

Tokenization object to hold tokenization, (processed text),and original text. Can hold tokenization as Ids or tokens. It also holds

Arguments: path: path to directory where array entries are concatenated into one big string file and the .len file are locate

BERT model ("Bidirectional Embedding Representations from a Transformer"). Params: config: a BertConfig class instance with the configura

Linear layer with column parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its second dimension as A = [A_1, ..

Linear layer with row parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its first dimension and X along its sec

Anneals the learning rate from start to zero along a cosine curve.

r""" :class:`transformers.BeamScorer` implementing standard beam search decoding. Adapted in part from `Facebook's XLM beam search code <

Class that manages dynamic loss scaling. It is recommended to use :class:`DynamicLossScaler` indirectly, by supplying ``dynamic_loss_scale

MLP for GPT2. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform gelu transformation, and project the

Parallel self-attention layer for GPT2. Self-attention layer takes input with size [b, s, h] where b is the batch size, s is the sequence len

Group of timers.

Processor for the X-Stance data set.

Runs basic tokenization (punctuation splitting, lower casing, etc.).

Configuration class to store the configuration of a `BertModel`.

Construct the embeddings from word, position and token_type embeddings.

Dataset containing sentencepairs for BERT training. Each index corresponds to a randomly generated sentence pair. Arguments: ds (Data

Loads a pretrained WordPiece tokenizer from `cache_dir` for tokenization in BERT training. Default to bert-large-uncased tokenizer.

Dataset to concatenate multiple datasets. Purpose: useful to assemble different existing datasets, possibly large-scale datasets as the c

Tracker for the cuda RNG states. Using the `add` method, a cuda rng state is initialized based on the input `seed` and is assigned to `name`.

:class:`FP16_Optimizer` is designed to wrap an existing PyTorch optimizer, and manage static or dynamic loss scaling and master weights in a

GLM Language model. The output of the forward method are the logits (parallel or serial depending on the `parallel_output` flag.

This class can be used to create a list of :class:`~transformers.LogitsProcessor` or :class:`~transformers.LogitsWarper` to subsequently proc

Class that manages a static loss scale. This class is intended to interact with :class:`FP16_Optimizer`, and should not be directly manipu

r""" :class:`transformers.LogitsProcessor` enforcing a min-length by setting EOS probability to 0. Args: min_length (:obj:`int`):

r""" :class:`transformers.LogitsProcessor` that enforces no repetition of n-grams. See `Fairseq <https://github.com/pytorch/fairseq/blob/a07cb

Parallel cross-attention layer for Transformer

A single layer transformer for GPT2. We use the following notation: h: hidden size n: number of attention heads b: batch

A single layer transformer for GPT2. We use the following notation: h: hidden size n: number of attention heads b: batch

callable class that runs a preprocessing, as well as tokenization step, on input text.

Dataset wrapper to access a subset of another dataset. Purpose: useful to index into existing datasets, possibly large-scale datasets as

Tokenizer object that handles text tokenization, command tokens, and type tokens. Command tokens and text tokens are stored together in o

Runs WordPiece tokenization.

Utility module that implements:: def forward(self, input): return input.half()

Processor for the AFQMC data set (CLUE version).

Processor for the AG news data set.

Processor for the AX-B diagnostic data set.

Processor for the AX-G diagnostic data set.

Abstract base class for all beam scorers that are used for :meth:`~transformers.PretrainedModel.beam_search` and :meth:`~transformers.Pretrai

BERT model with the masked language modeling head. This module comprises the BERT model followed by the masked language modeling head. Params

BERT model for multiple choice tasks. This module is composed of the BERT model with a linear layer on top of the pooled output. Params: