Send all values to device by calling v.to(device)

r""" # Using torch.hub ! import torch model = torch.hub.load('huggingface/transformers', 'model', 'bert-base-unca

r""" Instantiate a :class:`~transformers.ModelCard` from a pre-trained model model card. Parameters: pretrained_model_name_or_pat

Initialize and prunes weights if needed.

Creates a `tf.initializers.truncated_normal` with the given range. Args: initializer_range: float, initializer range for stddev. Retur

Updates attributes of this class with attributes from `config_dict`. Args: :obj:`Dict[str, any]`: Dictionary of

Deal with dynamic shape in tensorflow cleanly.

Save the provided data object with the representation for the current `DataFormat`. :param data: data to store :return:

Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing ``self.convert_tokens_to_ids(sel

Save the pipeline's model and tokenizer to the specified save_directory

r""" Generates sequences for models with a LM head. The method currently supports greedy decoding, beam-search decoding, sampling with temperature, sa

Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary.

Take the output of any QuestionAnswering head and will generate probalities for each span to be the actual answer. In additio

Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_i

Need to document this

Reads a tab separated value file.

Utility factory method to build a pipeline. Pipeline are made of: - A Tokenizer instance in charge of mapping raw textual input to

Function arguments can be inserted as boolean tensor and bool variables to cope with keras serialization we need to cast `output_atten

Returns a dictionary containing the encoded sequence or sequence pair and additional information: the mask for sequence classificatio

Converts a single index or a sequence of indices (integers) in a token " (resp.) a sequence of tokens (str), using the vocabulary and add

Returns a dictionary containing the encoded sequence or sequence pair and additional information: the mask for sequence classificatio

Get torch.device from module, assuming that the whole module has one device.

Create a schedule with a learning rate that decreases linearly after linearly increasing during a warmup period.

# Prepare head mask if needed # 1.0 in head_mask indicate we keep the head attention_probs has shape bsz x n_heads x N x N

One optimization step: forward of student AND teacher, backward on the loss (for gradient accumulation), and possibly a parameter upd

Constructs a `ModelCard` from a json file of parameters.

Only load the model if needed.

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

r""" # Using torch.hub ! import torch tokenizer = torch.hub.load('huggingface/transformers', 'tokenizer', 'bert-base-uncased'

The real training loop.

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Parse command-line args into instances of the specified dataclass types. This relies on argparse's `ArgumentParser.parse_known_args`

Add a new hypothesis to the list.

Main training entry point. Args: model_path: (Optional) Local path to model if model to train has been i

Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch. Padding side

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and retur

This will be True only in one process, even in distributed mode, even when training on multiple machines.

Cover moses empty string edge case. They return empty list for '' input!

Serializes this instance to a JSON string.

r""" Instantiates one of the base model classes of the library from a pre-trained model configuration. The `from_pretrained()` method

Arguments: src_texts: list of src language texts src_lang: default en_XX (english) tgt_texts: list of tgt

Prune a linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_grad=True.

Add a dictionary of special tokens (eos, pad, cls...) to the encoder and link them to class attributes. If special tokens are NOT in