Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may cont

Prediction/evaluation loop, shared by :obj:`Trainer.evaluate()` and :obj:`Trainer.predict()`. Works both with or without labels.

Prediction/evaluation loop, shared by :func:`~transformers.TFTrainer.evaluate` and :func:`~transformers.TFTrainer.predict`.

Prediction/evaluation loop, shared by :obj:`Trainer.evaluate()` and :obj:`Trainer.predict()`. Works both with or without labels.

Prediction/evaluation loop, shared by :obj:`Trainer.evaluate()` and :obj:`Trainer.predict()`. Works both with or without labels.

Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, tr

Implement in subclasses of :class:`~transfomers.PreTrainedModel` for custom behavior to prepare inputs in the generate method.

Implement in subclasses of :class:`~transfomers.TFPreTrainedModel` for custom behavior to prepare inputs in the generate method.

Build the relative positional vector between `pos` and `pooled_pos`.

Build the relative positional vector between `pos` and `pooled_pos`.

Resets the accumulated gradients on the current replica.

Resizes input token embeddings matrix of the model if :obj:`new_num_tokens != config.vocab_size`. Takes care of tying weights embedd

Computes the loss of the given features and labels pair. Subclass and override this method if you want to inject some custom behavio

Make sure that all the special tokens attributes of the tokenizer (:obj:`tokenizer.mask_token`, :obj:`tokenizer.cls_token`, etc.) are

Save the provided data object as a pickle-formatted binary data on the disk. Args: data (:obj:`dict` or list of :obj:`di

Save git information to output_dir/git_log.json

Go through the text data by order of src length with a bit of randomness. From fastai repo.

Converts a list of examples into a list of features that can be directly given as input to a model. It is model-dependant and takes advantage

Setup line-by-line tracing to record rss mem (RAM) at each line of a module or sub-module. See `./benchmark.py` for usage examples. Current

Stop memory tracing cleanly and return a summary of the memory trace if a trace is given. Args: - `memory_trace` (optional output of s

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (batch size, vocabula

Serializes this instance to a Python dictionary. Override the default `to_dict()` from `PretrainedConfig`. Returns: :obj

Serializes this instance to a Python dictionary. Override the default :meth:`~transformers.PretrainedConfig.to_dict`. Returns:

Serializes this instance while replace `Enum` by their values (for JSON serialization support).

Serializes this instance to a JSON string. Args: use_diff (:obj:`bool`, `optional`, defaults to :obj:`True`):

Convert self to a tuple containing all the attributes/keys that are not ``None``.

:param text: str :rtype: list(str) :return: a tokenized list of strings; concatenating this list returns\ the origina

Tokenizes a piece of text.

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (batch size, vocabula

Main training entry point. Args: model_path (:obj:`str`, `optional`): Local path to the model if the mod

Perform a training step on features and labels. Subclass and override to inject some custom behavior.

Perform a training step on a batch of inputs. Subclass and override to inject custom behavior. Args: model (:ob

Remove columns that are populated exclusively by pad_token_id

Truncates a sequence pair in-place following the strategy. Args: ids (:obj:`List[int]`): Tokenized input

Update config with summarization specific params.

Call HF API to know "whoami"

Constructs a MecabTokenizer. Args: **do_lower_case**: (`optional`) boolean (default True) Whether to lowercase th

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, tr

Cleans the padding history. Padding may be generated in two places when multiple conversations are provided as an input:

Computes global attention mask by putting attention on all tokens before `sep_token_id` if `before_sep_token is True` else after `sep_tok

Builds an input prepended by the history for this conversation, allowing multi-turn conversation with context

Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict. Overflowing tokens are converted to addit

-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]

Convert output_args_doc to display properly.