Performs a single optimization step. Arguments: closure (`Callable`, *optional*): A closure that reevaluates the model a

Creates an optimizer from its config with WarmUp custom object.

Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may cont

Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. Takes care of tying weights embeddings

Serializes this instance to a Python dictionary.

Set the random seed.

Filter dataset with heuristics. Config, test and has_no_keywords files are removed with a given probability.

Reads a tab separated value file.

Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().

Returns the model's input embeddings. Returns: `nn.Module`: A torch module mapping vocabulary to hidden states.

Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only goin

Returns the log level to be used depending on whether this process is the main process of node 0, main process of node non-0, or a no

Convert self to a tuple containing all the attributes/keys that are not `None`.

Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to tokenize - **return_ids**: Bo

Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch.

r""" Generates sequences of token ids for models with a language modeling head using **multinomial sampling** and can be used for text

Run pipeline tests for a specific `task` Args: task (`str`): A task name. This should be a key in the mapping `pi

Creates a draft of a model card using the information available to the `Trainer`. Args: language (`str`, *optional*):

Args: append_path: optional path to append to the tests dir path Return: The full path to the `tests` dir, so that the tests

Temporarily patches the variables defines in `tests_fetcher` to use a different location for the repo.

Loads `image` to a PIL Image. Args: image (`str` or `PIL.Image.Image`): The image to convert to the PIL Image format.

external allocation

Generate an image from the given prompts. If image_path is provided, the image is used as a starting point for the optimization. If im

Wraps a HuggingFace [`~datasets.Dataset`] as a `tf.data.Dataset` with collation and batching. This method is designed to create a "re

Constructs a `ModelCard` from a Python dictionary of parameters.

Return a copy of the `os.environ` object that sets up `PYTHONPATH` correctly, depending on the test suite it's invoked from. This is

Temporarily updates the `os.environ` dictionary in-place. Similar to mockenv The `os.environ` dictionary is updated in-place so that the mod

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Creates a repository in a temporary directory mimicking the structure of Transformers. Uses the list of models provided (which defaults to ju

Tests if `x` is a torch tensor or not. Safe to call even if torch is not installed.

Returns the number of added tokens when encoding a sequence with special tokens. <Tip> This encodes a dummy input and check

Transform pixel location to different reference.

Check if the model class is in supported by the pipeline. Args: supported_models (`List[str]` or `dict`):

Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named:

require_version wrapper which emits a core-specific hint on failure

Converts `image` to a numpy array. Optionally rescales it and puts the channel dimension as the first dimension. Args:

Creates a `tf.keras.initializers.TruncatedNormal` with the given range. Args: initializer_range (*float*, defaults to 0.02): Standar

Main function to quickly load a tool, be it on the Hub or in the Transformers library. Args: task_or_repo_id (`str`): Th

Catches the default value in the description of an argument inside a docstring and replaces it by the value passed. Args: descriptio

Returns an array containing the specified window. This window is intended to be used with `stft`. The following window types are supported:

Return a logger with the specified name. This function is not supposed to be directly accessed unless you are writing a custom transformers

Unified API to get any scheduler from its name. Args: name (`str` or `SchedulerType`): The name of the scheduler to use.

Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequ

Calculates a spectrogram over one waveform using the Short-Time Fourier Transform. This function can create the following kinds of spectrogr

Get the encoded token span corresponding to a word in a sequence of the batch. Token spans are returned as a [`~tokenization_utils_b

Add a new hypothesis to the list.

Infers the channel dimension format of `image`. Args: image (`np.ndarray`): The image to infer the channel dimension of.

Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned a

Evaluate a python expression using the content of the variables stored in a state and only evaluating a given set of functions. This fun