Deal with dynamic shape in tensorflow cleanly. Args: x (:obj:`tf.Tensor`): The tensor we want the shape of. Returns: :o

Creates a :obj:`tf.initializers.TruncatedNormal` with the given range. Args: initializer_range (`float`, defaults to 0.02): Standard

Send all values to device by calling :obj:`v.to(device)` (PyTorch only). Args: device (:obj:`str` or :obj:`torch.device`

r"""Instantiate a :class:`~transformers.ModelCard` from a pre-trained model model card. Parameters: pretrained_model_name_or_path

r""" # Using torch.hub ! import torch tokenizer = torch.hub.load('huggingface/transformers', 'tokenizer', 'bert-base-uncased'

Save the provided data object with the representation for the current :class:`~transformers.pipelines.PipelineDataFormat`. A

How the loss is computed by Trainer. By default, all models return the loss in the first element. Subclass and override for custom b

Take the output of any :obj:`ModelForQuestionAnswering` and will generate probalities for each span to be the actual answer.

Prune a linear layer to keep only entries in index. Used to remove heads. Args: layer (:obj:`torch.nn.Linear`): The layer to pr

Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args:

Reads a tab separated value file.

Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch.

Save the pipeline's model and tokenizer. Args: save_directory (:obj:`str`): A path to the directory wher

:obj:`tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer.

r""" # Using torch.hub ! import torch model = torch.hub.load('huggingface/transformers', 'model', 'bert-base-unca

r""" Examples:: >>> from transformers import AutoConfig, AutoModel >>> # Download model and configuration from S3 a

r""" Examples:: >>> from transformers import AutoConfig, AutoModel >>> # Download model and configuration from S3 a

r""" Generates sequences for models with a language modeling head. The method currently supports greedy decoding, beam-search decoding

Performs a single optimization step. Arguments: closure (:obj:`Callable`, `optional`): A closure that reevaluates the mo

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing ``self.convert_tokens_to_ids(sel

Set requires_grad=False for each of model.parameters()

Prepare the head mask if needed. Args: head_mask (:obj:`torch.Tensor` with shape :obj:`[num_heads]` or :obj:`[num_hidden

Constructs a `ModelCard` from a json file of parameters.

Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only goin

Deal with dynamic shape in tensorflow cleanly.

Finds the heads and their indices taking :obj:`already_pruned_heads` into account. Args: heads (:obj:`List[int]`): List of the indic

splits sequence length dim of vectors into `dim_factor_1` and `dim_factor_2` dims

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and retur

Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_i

This function chunks the :obj:`input_tensors` into smaller input tensor parts of size :obj:`chunk_size` over the dimension :obj:`chunk_dim`.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the

Serializes this instance to a JSON string.

Sanitized serialization to use with TensorBoard’s hparams

Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only goin

Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only goin

Log :obj:`logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args

Log :obj:`logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args

Cover moses empty string edge case. They return empty list for '' input!

Helper function for reproducible behavior to set the seed in ``random``, ``numpy``, ``torch`` and/or ``tf`` (if installed). Args:

Add a new hypothesis to the list.

Function arguments can be inserted as boolean tensor and bool variables to cope with Keras serialization we need to cast the bool argumnets (

Prepare :obj:`inputs` before feeding them to the model, converting them to tensors if they are not already and handling potential sta

Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (