Deal with dynamic shape in tensorflow cleanly. Args: tensor (`tf.Tensor` or `np.ndarray`): The tensor we want the shape of. Ret

Finds the heads and their indices taking `already_pruned_heads` into account. Args: heads (`List[int]`): List of the indices of head

A context manager for torch distributed environment where on needs to do something on the main process, while blocking replicas, and

Computes the output length of the convolutional layers

Upload the {object_files} to the 🤗 Model Hub while synchronizing a local clone of the repo in `repo_path_or_name`. Parameter

Parse command-line args into instances of the specified dataclass types. This relies on argparse's `ArgumentParser.parse_known_args`

The index of the current process used.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Stable wrapper that returns the same output as `tf.nn.softmax`, but that works reliably with XLA on CPU. It is meant as a workaround for the

Add a new hypothesis to the list.

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Returns the (height, width) dimensions of the image. Args: image (`np.ndarray`): The image to get the dimensions of.

Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. Takes care of tying weights embeddings

r""" Instantiate an encoder and a decoder from one or two base classes of the library from pretrained model checkpoints. The

Generate the mapping between supported the features and their corresponding OnnxConfig for a given model. Args: *supported_features:

Checks whether `char` is a whitespace character.

Returns the number of added tokens when encoding a sequence with special tokens. <Tip> This encodes a dummy input and check

Alternative helper method that does not use `argparse` at all, instead loading a json file and populating the dataclass types.

Converts `image` to the channel dimension format specified by `channel_dim`. Args: image (`numpy.ndarray`): The image to

Perform a runtime check of the dependency versions, using the exact same syntax used by pip. The installed module version comes from the *si

Sends telemetry that helps tracking the examples use. Args: example_name (`str`): The name of the example. *example_args (da

The real training loop.

Checks if `torch_xla` is installed and potentially if a TPU is in the environment

Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT

Args: append_path: optional path to append to the tests dir path Return: The full path to the `tests` dir, so that the tests

Checks whether `char` is a punctuation character.

Converts bounding boxes from center format to corners format. center format: contains the coordinate for the center of the box and its width

Closes the progress bar.

Dict of bias attached to an LM head. The key represents the name of the bias attribute. Return: `tf.Variable`: The weigh

Will save the model, so you can reload it using `from_pretrained()`. Will only save from the main process.

Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().

Creates an optimizer from its config with WarmUp custom object.

Ensure that the input is a list of images. If the input is a single image, it is converted to a list of length 1. If the input is a batch of

Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are ta

Invert an attention mask (e.g., switches 0. and 1.). Args: encoder_attention_mask (`torch.Tensor`): An attention mask.

Checks whether `char` is a control character.

Whether or not the current process is the local process, based on `xm.get_ordinal()` (for TPUs) first, then on `local_rank`.

Tests if `x` is a torch tensor or not. Safe to call even if torch is not installed.

One optimization step: forward of student AND teacher, backward on the loss (for gradient accumulation), and possibly a parameter upd

Rescales `image` by `scale`. Args: image (`np.ndarray`): The image to rescale. scale (`float`): The

Tries to locate a file in a local folder and repo, downloads and cache it if necessary. Args: path_or_repo_id (`str` or `os.PathLike

Constructs a `ModelCard` from a json file of parameters.

Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: word_attention_m

From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a [`PretrainedConfig`] us

A function that performs integer division across different versions of PyTorch.

Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may cont

Traces `root` and returns the corresponding FX `torch.fx.Graph` representation. `root` can either be a `torch.nn.Module` instance or

Reads a tab separated value file.