Loads a data file into a list of `InputBatch`s.

Relies on the assumption that there are no language codes like pt_br in models that are not in GROUP_TO_OPUS_NAME.

Convert a TF 2.0 model variable name in a pytorch model weight name. Conventions for TF2.0 scopes -> PyTorch attribute names conversions:

Creates an optimizer with learning rate schedule.

Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified f

We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. :param torch.Tensor

Call HF API to delete a file stored by user

Delete token. Do not fail if token does not exist.

Ensure input are presented in the correct order, without any None Args: model: The model used to forward the input data token

r""" Instantiates one of the base model classes of the library from a configuration. Note: Loading a model from its confi

r""" Instantiate a :class:`~transformers.EncoderDecoderConfig` (or a derived class) from a pre-trained encoder model configuration and decoder

Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For infor

From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a Config using `from_dict

measures current cpu memory usage of a given `process_id` Args: - `process_id`: (`int`)

Project the tokenized prediction back to the original text.

Project the tokenized prediction back to the original text.

Select framework (TensorFlow/PyTorch) to use. If both frameworks are installed and no specific model is provided, defaults to using PyTorch.

Returns the model's input embeddings. Returns: :obj:`nn.Module`: A torch module mapping vocabulary to hi

Returns the model's output embeddings. Returns: :obj:`nn.Module`: A torch module mapping hidden states t

Computes the exact and f1 scores from the examples and the model predictions

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the

Loads a data file into a list of ``InputFeatures`` Args: examples: List of ``InputExamples`` containing the examples. tokeni

Init decoder state

Is the configuration aligned with the MBART paper.

Call HF API to list all stored files for user (or one of their organizations).

Load pytorch checkpoints in a TF 2.0 model

Load pytorch state_dict in a TF 2.0 model.

Loads a vocabulary file into a dictionary.

measures peak cpu memory consumption of a given `function` running the function for at least interval seconds and at most 20

Helper to get num of examples from a DataLoader, by accessing its Dataset.

Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the dataclass types.