Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch` and/or `tf` (if installed). Args: seed (`int

Returns the maximum sequence length of the cached states, if there is any.

Tokenize and prepare for the model a sequence or a pair of sequences. <Tip warning={true}> This method is deprecated, `__ca

Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation. The special key `""`

Constructs a `ModelCard` from a json file of parameters.

Returns an array containing the specified window. This window is intended to be used with `stft`. The following window types are supported:

Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are ta

Check if `torch_xla` is available. To train a native pytorch job in an environment with torch xla installed, set the USE_TORCH_XLA to false.

Rescores a prediction based on penalty functions

Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then compu

One optimization step: forward of student AND teacher, backward on the loss (for gradient accumulation), and possibly a parameter upd

Whether or not the current process is the local process, based on `xm.get_ordinal()` (for TPUs) first, then on `local_rank`.

Send all values to device by calling `v.to(*args, **kwargs)` (PyTorch only). This should support casting in different `dtypes` and se

Checks whether `char` is a punctuation character.

Tests if `x` is a torch tensor or not. Safe to call even if torch is not installed.

Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: word_attention_m

Pads the `image` with the specified (height, width) `padding` and `mode`. Args: image (`np.ndarray`): The image to pad.

Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may cont

Performs a single optimization step. Arguments: closure (`Callable`, *optional*): A closure that reevaluates the model a

Traces `root` and returns the corresponding FX `torch.fx.Graph` representation. `root` can either be a `torch.nn.Module` instance or

Deal with dynamic shape in tensorflow cleanly. Args: tensor (`tf.Tensor` or `np.ndarray`): The tensor we want the shape of. Ret

Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input conversations to turn them i

Set the random seed.

Serializes this instance to a Python dictionary. Override the default [`~PretrainedConfig.to_dict`]. Returns: `Dict[str,

Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequ

Checks whether `char` is a control character.

Resizes `image`. Enforces conversion of input to PIL.Image. Args: image (`PIL.Image.Image` or `np.ndarray` or `torch.Ten

Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length

Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. Args: batch_text_or_text_pairs (

Reads a tab separated value file.

Load flax checkpoints in a PyTorch model

Analogous to the to() method of torch Tensors Args: device: A torch device dtype:

Attend to all tokens in masked rows from the expanded attention mask, for example the relevant first rows when using left padding. Th

Overrides the `from_dict` method from the base class to save support of deprecated `reduce_labels` in old configs

Constructs an identity transformation. Args: shape: The desired shape dtype:

r""" Args: batch_size (`int`): batch_size used for fast auto-regressive decoding. Defines the batch size of the in

Get number of (optionally, trainable or non-embeddings) parameters in the module. Args: only_trainable (`bool`, *optiona

Returns the log level to be used depending on whether this process is the main process of node 0, main process of node non-0, or a no

This class computes the losses using the class predictions, mask predictions and the contrastive queries. Oneformer calculates the c