Send all values to device by calling :obj:`v.to(device)` (PyTorch only). Args: device (:obj:`str` or :obj:`torch.device`

r""" # Using torch.hub ! import torch model = torch.hub.load('huggingface/transformers', 'model', 'bert-base-unca

Deal with dynamic shape in tensorflow cleanly. Args: x (:obj:`tf.Tensor`): The tensor we want the shape of. Returns: :o

Creates a :obj:`tf.initializers.TruncatedNormal` with the given range. Args: initializer_range (`float`, defaults to 0.02): Standard

Save the pipeline's model and tokenizer. Args: save_directory (:obj:`str`): A path to the directory wher

Need to document this

r""" Generates sequences for models with a language modeling head. The method currently supports greedy decoding, multinomial sampling

Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_ids**: Bo

Convert list of tokens to ids Args: tokens (:obj:`list<str>`): list of tokens Returns: List of ids

Tokenize and prepare for the model a sequence or a pair of sequences. .. warning:: This method is deprecated, ``__call__

r""" # Using torch.hub ! import torch tokenizer = torch.hub.load('huggingface/transformers', 'tokenizer', 'bert-base-uncased'

Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. .. warning:: This method is depr

Main training entry point. Args: model_path (:obj:`str`, `optional`): Local path to the model if the mod

Save the provided data object with the representation for the current :class:`~transformers.pipelines.PipelineDataFormat`. A

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

Prune a linear layer to keep only entries in index. Used to remove heads. Args: layer (:obj:`torch.nn.Linear`): The layer to pr

Utility factory method to build a :class:`~transformers.Pipeline`. Pipelines are made of: - A :doc:`tokenizer <tokenizer>` in charg

Convert list of ids to tokens Args: ids (:obj:`list<int>`): list of ids Returns: List of tokens

r""" Instantiate a :class:`~transformers.ModelCard` from a pre-trained model model card. Parameters: pretrained_model_nam

Args: inputs (`torch.FloatTensor`) The input matrix from which the binarizer computes the binary mask.

Args: tmp_dir (:obj:`string`, `optional`): use this path, if None a unique path will be assigned befo

Reads a tab separated value file.

:obj:`torch.device`: The device on which the module is (assuming that all the module parameters are on the same device).

Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch.

Log :obj:`logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args

Add a new hypothesis to the list.

Convert self to a tuple containing all the attributes/keys that are not ``None``.

Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args

Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are ta

copied from transformers.modeling_tf_utils

:obj:`tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer.

Parse command-line args into instances of the specified dataclass types. This relies on argparse's `ArgumentParser.parse_known_args`

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Get number of (optionally, trainable or non-embeddings) parameters in the module. Args: only_trainable (:obj:`bool`, `op

Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only goin

Performs a single optimization step. Arguments: closure (:obj:`Callable`, `optional`): A closure that reevaluates the mo

r""" Examples:: >>> from transformers import AutoConfig, AutoModel >>> # Download model and configuration from S3 a

Prepare the head mask if needed. Args: head_mask (:obj:`torch.Tensor` with shape :obj:`[num_heads]` or :obj:`[num_hidden

One optimization step: forward of student AND teacher, backward on the loss (for gradient accumulation), and possibly a parameter upd

Constructs a `ModelCard` from a json file of parameters.

Resizes input token embeddings matrix of the model if :obj:`new_num_tokens != config.vocab_size`. Takes care of tying weights embedd

r""" Examples:: >>> from transformers import AutoConfig, AutoModel >>> # Download model and configuration from S3 a

Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may cont

Will save the model, so you can reload it using :obj:`from_pretrained()`. Will only save from the world_master process (unless in TP

Checks whether `char` is a whitespace character.

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and return th