Checks whether `chars` is a control character.

Checks whether `chars` is a whitespace character.

Given the sequence length of the new inputs, returns the usable length of the cache.

Wraps a HuggingFace [`~datasets.Dataset`] as a `tf.data.Dataset` with collation and batching. This method is designed to create a "re

Unified API to get any scheduler from its name. Args: name (`str` or `SchedulerType`): The name of the scheduler to use.

Framework-agnostic version of `numpy.reshape` that will work on torch/TensorFlow/Jax tensors as well as NumPy arrays.

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the gi

Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one.

Converts a token (str) in an id using the vocab.

Depending on the input and internal state we might trigger a warning about a sequence that is too long for its corresponding model

Creates a repository in a temporary directory mimicking the structure of Transformers. Uses the list of models provided (which defaults to ju

Return a logger with the specified name. This function is not supposed to be directly accessed unless you are writing a custom transformers

Tests if `x` is a tensorflow tensor or not. Safe to call even if tensorflow is not installed.

Load adapter weights from file or remote Hub folder. If you are not familiar with adapters and PEFT methods, we invite you to read mo

This is a context manager to temporarily change transformers modules logging level to the desired value and have it restored to the original

Args: in_dim: The final dimension of inputs to the layer out_dim: The final dimension

Creates an index. Args: indices: <int32> Tensor of indices, same shape as `values`. num_segments: <int32> Scalar

Args: inputs (`torch.FloatTensor`) The input matrix from which the binarizer computes the binary mask.

Center crop an image to a certain size. Args: image (`np.ndarray`): Image to center crop. cr

Converts a sequence of tokens (string) in a single string.

r""" Example: ```python >>> from transformers import VisionEncoderDecoderModel, AutoImageProcessor, AutoTokenizer >>>

This function generates a JSON schema for a given function, based on its docstring and type hints. This is mostly used for passing lists of t

Returns `True` if index is already initialized.

Just EOS

When used in normal mode, this method forwards all its arguments to MCTCTFeatureExtractor's [`~MCTCTFeatureExtractor.pad`] and return

Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped

Saves the attributes of this processor (tokenizer...) in the specified directory so that it can be reloaded using the [`~BarkProcesso

Converts `image` to a numpy array. Optionally rescales it and puts the channel dimension as the first dimension. Args:

Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`

Check if the model class is in supported by the pipeline. Args: supported_models (`List[str]` or `dict`):