During generation, decide whether to pass the `past` variable to the next forward pass.

Add a new hypothesis to the list.

Build shared word embedding layer

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms. Args: out

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

Convert a TF 2.0 model variable name in a pytorch model weight name. Conventions for TF2.0 scopes -> PyTorch attribute names conversions:

Converts a sequence of token ids in a single string. The most simple way to do it is ``" ".join(tokens)`` but we often want to remov

QuestionAnsweringPipeline leverages the :class:`~transformers.SquadExample` internally. This helper method encapsulate all the logic

Create a mask from the two sequences passed to be used in a sequence-pair classification task. An XLM sequence pair mask has the foll

Detokenize the provided tokens ids to readable text: - **tokens_ids**: List of tokens ids - **skip_special_tokens**: Flag ind

Only used by LegacyDataset

Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are

Input shape: Time(SeqLen) x Batch x Channel

r""" Instantiate a :class:`~transformers.EncoderDecoderConfig` (or a derived class) from a pre-trained encoder model configuration and decoder

r""" Instantiate a :class:`~transformers.EncoderDecoderConfig` (or a derived class) from a pre-trained encoder model configuration and decoder

Append a string-identifier at the end (before the extension, if any) to the provided filepath. Args: filename: pathlib.Path The actu

measures current cpu memory usage of a given `process_id` Args: - `process_id`: (`int`) process_id

See base class.

Project the tokenized prediction back to the original text.

Select framework (TensorFlow or PyTorch) to use. Args: model (:obj:`str`, :class:`~transformers.PreTrainedModel` or :class:`~transfo

Returns the model's input embeddings. Returns: :obj:`tf.keras.layers.Layer`: A torch module mapping vocabulary to hidden

Gets the list of labels for this data set.

Generate hidden states mask, and optionally an attention mask.

Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings)

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings)

Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings)

See base class.

Returns the vocabulary as a dictionary of token to index. :obj:`tokenizer.get_vocab()[token]` is equivalent to :obj:`tokenizer.conve

Loads a data file into a list of ``InputFeatures`` Args: examples: List of ``InputExamples`` or ``tf.data.Dataset`` containing the e

Attempt to infer the static vs dynamic axes for each input and output tensors for a specific model. Args: nlp: The pipeline object ho

Returns the attention inputs associated to the inputs of the model.

Returns the attention inputs associated to the inputs of the model.

Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only goin

Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only goin

From fairseq

Load pytorch checkpoints in a TF 2.0 model

Load pytorch state_dict in a TF 2.0 model.

Loads a vocabulary file into a dictionary.

Log :obj:`logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args

measures peak cpu memory consumption of a given `function` running the function for at least interval seconds and at most 20 * interva