Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at the en

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

splits sequence length dim of vectors into `dim_factor_1` and `dim_factor_2` dims

Builds the mask. The attention mechanism will only attend to positions with value 1.

Prediction/evaluation loop, shared by `evaluate()` and `predict()`.

Creates a random float32 tensor

See base class.

See base class.

Turns 1->0, 0->1, False->True, True-> False

Call HF API to sign in a user and get a token if credentials are valid. Outputs: token if credentials are valid

Get the public list of all the models on huggingface, including the community models

Extract the story and summary from a story file. Arguments: raw_story (str): content of the story file as an utf-8 encoded string.

Resets the accumulated gradients on the current replica.

Save the pretrained model.

This function wraps another function into its own separated process. In order to ensure accurate memory measurements it is important

Save this instance to a json file.

Train method to train the model.

Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().

Adapt the source and target sequences' lengths to the block size. If the sequence is shorter we append padding token to the right of the sequence

Runs basic whitespace cleaning and splitting on a piece of text.

TFConv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2) Basically works like a Linear layer but the weights a

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Creates examples for the training and dev sets.

Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length) an

Used to implement attention between consecutive chunks. Args: vectors: array of shape [batch_size, num_attention_heads,

Prepare masks that ignore padding tokens in the decoder and a causal mask for the decoder if none are provided. This mimics the default behavior i

This method shows how to compute: - head attention entropy - head importance scores according to http://arxiv.org/abs/1905.10650

XLNet write prediction logic (more complex than Bert's). Write final predictions to the json file and log-odds of null if needed. Re

Write final predictions to the json file and log-odds of null if needed.

Loads a data file into a list of `InputFeatures`

Converts a sequence of tokens (string) in a single string.

Very simple data collator that: - simply collates batches of dict-like objects - Performs special handling for potential keys named:

Creates a list of :class:`~transformers.data.processors.squad.SquadExample` using a TFDS dataset. Args: dataset: The tfd

Generate hidden states mask, and optionally an attention mask.

Loads a data file into a list of ``InputFeatures`` Args: examples: List of ``InputExamples`` or ``tf.data.Dataset`` containing the e

Lower text and remove punctuation, articles and extra whitespace.

Prune a Conv1D layer (a model parameters) to keep only entries in index. A Conv1D work as a Linear layer (see e.g. BERT) but the weights are