Performs invalid character removal and whitespace cleanup on text.

Initialize the weights.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTraine

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Adds whitespace around any CJK character.

A map of modules from TF to PyTorch. This time I use a map to keep the PyTorch model as identical to the original PyTorch model as possible.

A map of modules from TF to PyTorch. I use a map to keep the PyTorch model as identical to the original PyTorch model as possible.

Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code bar

cache hidden states into memory.

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.

Creates causal attention mask. Float mask where 1.0 indicates masked, 0.0 indicates not-masked. Args: qlen: TODO Lysandr

Constructs a `Config` from a Python dictionary of parameters.

Constructs a `BertConfig` from a Python dictionary of parameters.

Constructs a `BertConfig` from a json file of parameters.

GELU activation https://arxiv.org/abs/1606.08415 https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/model_pytorch.p

See base class.

Given a URL, look for the corresponding dataset in the local cache. If it's not there, download it. Then return the path to the cached file.

Generate hidden states mask, and optionally an attention mask.

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the

See base class.

See base class.

Example: >>> init_logger(log_file) >>> logger.info("abc'")

Load tf checkpoints in a pytorch model.

Load tf checkpoints in a pytorch model.

Load tf checkpoints in a pytorch model

Loads a vocabulary file into a dictionary.

r""" Computes log probabilities for all :math:`n\_classes` From: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/adaptive.py

Lowercase and strips accents from a piece of text based on https://github.com/facebookresearch/XLM/blob/master/tools/lowercase_and_remove_acc

Returns the number of added tokens when encoding a sequence with special tokens. Note: This encodes inputs and checks th

setup GPU device if available, move model into configured device # 如果n_gpu_use为数字，则使用range生成list # 如果输入的是一个list，则默认使用list[0]作为controll

Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, tr

Prunes heads of the base model. Arguments: heads_to_prune: dict with keys being selected layer indices (`int`) and asso

create relative positional encoding.

Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/remove-non-printing-char.perl

Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/replace-unicode-punctuation.perl

Sennrich's WMT16 scripts for Romanian preprocessing, used by model `xlm-mlm-enro-1024`

Check ETag on S3 object.

Pull a file directly from S3.

labels: [b1, b2] Return true_log_probs: [b1, b2] samp_log_probs: [n_sample] neg_samples: [n_s

embedding: an nn.Embedding layer bias: [n_vocab] labels: [b1, b2] inputs: [b1, b2, n_emb] sampler: you may us

Save the tokenizer vocabulary and merge files to a directory.

Save the tokenizer vocabulary to a directory. This method does *NOT* save added tokens and special token mappings. Please us

fixes some issues the spacy tokenizer had on books corpus also does some whitespace standardization

Make sure we are sharing the input and output embeddings. Export to TorchScript can't handle parameter sharing so we are cloning them ins

Make sure we are sharing the embeddings

Make sure we are sharing the embeddings

Run this to be sure output and input (adaptive) softmax weights are tied