See base class.

Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (

Prepare the head mask if needed. Args: head_mask (`torch.Tensor` with shape `[num_heads]` or `[num_hidden_layers x num_h

Returns the indent in given line (as string).

Returns the module name corresponding to a module file.

Returns the first found floating dtype in parameters if there is one, otherwise returns the last dtype it found.

Get all [test] classes in a model test file with attribute `all_model_classes` that are non-empty. These are usually the (model) test classes con

Returns the test [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior.

Returns the training [`~torch.utils.data.DataLoader`]. Will use no sampler if `train_dataset` does not implement `__len__`, a random

See base class.

r""" Generates sequences of token ids for models with a language modeling head using **greedy decoding** and can be used for text-deco

r""" Generates sequences of token ids for models with a language modeling head using **diverse beam search decoding** and can be used

Creates a random int32 tensor of the shape within the vocab size.

Infers the framework of a given model without using isinstance(), because we cannot guarantee that the relevant classes are imported or avail

Select framework (TensorFlow or PyTorch) to use from the `model` passed. Returns a tuple (framework, model). If `model` is instantiated, thi

If needed prunes and maybe initializes weights. If using a custom `PreTrainedModel`, you need to implement any initialization logic i

Check if the code commented as a copy in a file matches the original. Args: filename (`str`): The name of the file to ch

Reads a PyTorch checkpoint file, returning properly formatted errors if they arise.

Return item located at `loader_batch_index` within the current `loader_batch_data`.

Lower text and remove punctuation, articles and extra whitespace.

Lower text and remove punctuation, articles and extra whitespace.

Lower text and remove punctuation, articles and extra whitespace.

Event called after an evaluation phase.

Normalization on the loss (gradient accumulation or distributed training), followed by backward pass on the loss, possibly followed b

Pad input values / input vectors or a batch of input values / input vectors up to predefined length or to the max sequence length in

Remove a callback from the current list of [`~transformers.TrainerCallback`] and returns it. If the callback is not found, returns `

Preprocess will take the `input_` of a specific pipeline and return a dictionary of everything necessary for `_forward` to run proper

Extract the story and summary from a story file. Arguments: raw_story (str): content of the story file as an utf-8 encoded string. R

Retrieve the model classes associated to a given model. Args: model_type (`str`): A valid model type (like "bert" or "gpt2")

Save only the vocabulary of the tokenizer (vocabulary + added tokens). This method won't save the configuration and special token ma

This function wraps another function into its own separated process. In order to ensure accurate memory measurements it is important that the

If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT official documentation: htt

A helper function to set a given tensor (parameter of buffer) of a module on a specific device (note that doing `param.to(device)` creates a

For reproducible training Args: seed: A seed for reproducible training

Set the verbosity level for the 🤗 Transformers's root logger. Args: verbosity (`int`): Logging level, e.g., one of:

Splits a model state dictionary in sub-checkpoints so that the final size of each sub-checkpoint does not exceed a given size. The sub-c

Simplify a list of replacement patterns to make sure there are no needless ones. For instance in the sequence "Bert->BertNew, BertConfig->Be

r""" Helper to map inputs to the positive orthant by applying the square-plus operation. Reference: https://twitter.com/jon_barron/sta

stop tracking for the passed stage

combine stop and metrics update in one call for simpler code

Splits a model state dictionary in sub-checkpoints so that the final size of each sub-checkpoint does not exceed a given size. The sub-c

r""" Cast the floating-point `params` to `jax.numpy.bfloat16`. This returns a new `params` tree and does not cast the `params` in plac

Removes all attributes from config which correspond to the default config attributes for better readability and serializes to a Pytho

Returns the stringified version of that object. In the case of an AgentImage, it is a path to the serialized version of the image.

Converts a string in a sequence of tokens, using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-b

Remove columns that are populated exclusively by pad_token_id

Remove columns that are populated exclusively by pad_token_id

Adapt the source and target sequences' lengths to the block size. If the sequence is shorter we append padding token to the right of the sequence.

Update config with summarization specific params.

Write out the message in Line delimited JSON.