Initialize and prunes weights if needed.

Reads a tab separated value file.

r""" Instantiates one of the base model classes of the library from a pre-trained model configuration. The model class to instantiate

Prune a linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_grad=True.

Tie or clone module weights depending of weither we are using TorchScript or not

Converts a single token, or a sequence of tokens, (str/unicode) in a single integer id (resp. a sequence of ids), using the vocabulary.

Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at the en

Conv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2) Basically works like a Linear layer but the weights are

Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args:

Reads a tab separated value file.

Converts a sequence of ids (integer) in a string, using the tokenizer and vocabulary with options to remove special tokens and clean

Prune a Conv1D layer (a model parameters) to keep only entries in index. A Conv1D work as a Linear layer (see e.g. BERT) but the weights are

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and retur

Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing ``self.convert_tokens_to_ids(sel

Constructs a `BertConfig` from a json file of parameters.

Core relative positional attention operations.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

See base class.

Post-attention processing.

Save a model and its configuration file to a directory, so that it can be re-loaded using the `:func:`~transformers.PreTrainedModel.from_

Make sure we are sharing the input and output embeddings. Export to TorchScript can't handle parameter sharing so we are cloning them

Runs basic whitespace cleaning and splitting on a piece of text.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Tie or clone module weights depending of weither we are using TorchScript or not

Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-wo

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Converts a single index or a sequence of indices (integers) in a token " (resp.) a sequence of tokens (str/unicode), using the vocabulary

Converts a sequence of tokens (string) in a single string. The most simple way to do it is ' '.join(self.convert_ids_to_tokens(token_ids)

Returns a dictionary containing the encoded sequence or sequence pair and additional informations: the mask for sequence classificati

Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings)

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings)

设置整个开发环境的seed :param seed: :param device: :return:

Split a full s3 path into the bucket name and path.

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

Serializes this instance to a JSON string.

Serializes this instance to a JSON string.

Reference : https://github.com/tensorflow/tensorflow/blob/r1.10/tensorflow/python/ops/candidate_sampling_ops.py `P(class) = (log(