Convert the inner content to tensors. Args: tensor_type (:obj:`str` or :class:`~transformers.tokenization_utils_base.Ten

Converts `text` to Unicode (if it's not already), assuming UTF-8 input.

Creates examples for a single document.

Creates examples for a single document.

Creates examples for a single document.

Creates causal attention mask. Float mask where 1.0 indicates masked, 0.0 indicates not-masked. Args: qlen: Sequence len

Creates causal attention mask. Float mask where 1.0 indicates masked, 0.0 indicates not-masked. Args: qlen: TODO Lysandr

Creates an optimizer with a learning rate schedule using a warmup phase followed by a linear decay. Args: init_lr (:obj:`float`):

Setup the optimizer and the learning rate scheduler. We provide a reasonable default that works well. If you want to use something e

Setup the optimizer and the learning rate scheduler. We provide a reasonable default that works well. If you want to use something e

Setup the optimizer and the learning rate scheduler. We provide a reasonable default that works well. If you want to use something e

Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified fr

Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified fr

Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modifie

Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modifie

We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. :param tf.Tensor input

We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. :param tf.Tensor input

We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. :param torch.Tensor i

We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. :param torch.Tensor i

Create the token type IDs corresponding to the sequences passed. `What are token type IDs? <../glossary.html#token-type-ids>`__

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

Call HF API to delete a file stored by user

Delete token. Do not fail if token does not exist.

Inverts the operation of `tokenize_numbers`. This is replacing ' @,@ ' and ' @.@' by ',' and '.'. Args: text: A string where the

Enforce the repetition penalty (from the `CTRL paper <https://arxiv.org/abs/1909.05858>`__).

Ensure input are presented in the correct order, without any None Args: model: The model used to forward the input data token

Helper function to estimate the total number of tokens from the model inputs. Args: inputs (:obj:`dict`): The model inpu

Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are

FP16-compatible function that fills a input_ids with -inf.

FP16-compatible function that fills a input_ids with -inf.

For models that inherit from :class:`~transformers.PretrainedModel`, uses that method to compute the number of floating point operati

For models that inherit from :class:`~transformers.PretrainedModel`, uses that method to compute the number of floating point operati

Input shape: Time(SeqLen) x Batch x Channel

Freeze token embeddings and positional embeddings for bart, just token embeddings for t5.

Instantiates a :class:`~transformers.PretrainedConfig` from a Python dictionary of parameters. Args: config_dict (:obj:`

Creates an instance of the right subclass of :class:`~transformers.pipelines.PipelineDataFormat` depending on :obj:`format`.

Implements RAG token decoding. Args: input_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `

Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: :obj:`Dict[str, int]`: The added

Saves the best model by validation ROUGE2 score.

Returns a list of dictionaries, containing titles and text of the retrieved documents. Args: doc_ids (:obj:`np.ndarray`

Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3

Returns the evaluation :class:`~torch.utils.data.DataLoader`. Will use no sampler if :obj:`self.eval_dataset` is a :obj:`torch.utils

Returns the evaluation :class:`~torch.utils.data.DataLoader`. Will use no sampler if :obj:`self.eval_dataset` is a :obj:`torch.utils

Returns the evaluation :class:`~torch.utils.data.DataLoader`. Will use no sampler if :obj:`self.eval_dataset` is a :obj:`torch.utils

Returns the evaluation :class:`~tf.data.Dataset`. Args: eval_dataset (:class:`~tf.data.Dataset`, `optional`):

See base class.

Given a URL, look for the corresponding file in the local cache. If it's not there, download it. Then return the path to the cached file.

See base class.

Generate hidden states mask, and optionally an attention mask.

Generate hidden states mask, and optionally an attention mask.

Returns the model's output embeddings. Returns: :obj:`nn.Module`: A torch module mapping hidden states to vocabulary.