Builds the mask. The attention mechanism will only attend to positions with value 1.

Uses sacrebleu's corpus_bleu implementation.

This function computes individual parts of the relative position buckets. For more detail, see paper.

Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the foll

Detokenize the provided tokens ids to readable text: - **tokens_ids**: List of tokens ids - **skip_special_tokens**: Flag indicating

r""" Instantiate a :class:`~transformers.EncoderDecoderConfig` (or a derived class) from a pre-trained encoder model configuration and

See base class.

Returns a list of dictionaries, containing titles and text of the retrieved documents. Args: doc_ids (:obj:`np.ndarray`

Return a copy of the ``os.environ`` object that sets up ``PYTHONPATH`` correctly, depending on the test suite it's invoked from. This

Creates a list of :class:`~transformers.data.processors.squad.SquadExample` using a TFDS dataset. Args: dataset: The tfd

See base class.

Group together the adjacent tokens with the same entity predicted. Args: entities (:obj:`dict`): The entities predicted

Retriever initialization function, needs to be called from the training process. The function sets some common parameters and environ

Turns 1->0, 0->1, False->True, True-> False

Turns 1->0, 0->1, False->True, True-> False

If there are enough hypotheses and that none of the hypotheses being generated can become better than the worst one in the heap, then

Create an instance from the content of :obj:`json_path`.

Call HF API to sign in a user and get a token if credentials are valid. Outputs: token if credentials are valid Throws: req

Lower text and remove punctuation, articles and extra whitespace.

Lower text and remove punctuation, articles and extra whitespace.

pickle.dump(obj, path)

Apply 1D pooling to a tensor of size [B x T (x H)].

Apply 1D pooling to a tensor of size [B x T (x H)].

Post-attention processing.

Implement in subclasses of :class:`~transformers.PreTrainedModel` for custom behavior to prepare inputs in the generate method.

Extract the story and summary from a story file. Arguments: raw_story (str): content of the story file as an utf-8 encoded string. R

Core relative positional attention operations.

Core relative positional attention operations.

Resets the accumulated gradients on the current replica.

Build a resized question answering linear layer Module from a provided new linear layer. Increasing the size will add newly initializ

This function wraps another function into its own separated process. In order to ensure accurate memory measurements it is important that the

Helper function for reproducible behavior to set the seed in ``random``, ``numpy``, ``torch`` and/or ``tf`` (if installed). Args:

Serializes this instance to a Python dictionary. Returns: :obj:`Dict[str, Any]`: Dictionary of all the attributes that m

Save this instance to a json file.

Converts a string in a sequence of tokens, using the tokenizer. Note that, unlike Fast tokenizers (instances of PreTrainedTokenizerF

Remove columns that are populated exclusively by pad_token_id

Adapt the source and target sequences' lengths to the block size. If the sequence is shorter we append padding token to the right of the sequence.

Runs basic whitespace cleaning and splitting on a piece of text.

Converts an index (integer) in a token (str) using the vocab.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training, dev and test sets.

Creates examples for the training and dev sets.