This subclass of `argparse.ArgumentParser` uses type hints on dataclasses to generate arguments. The class is designed to play well with the

TrainingArguments is the subset of the arguments we use in our example scripts **which relate to the training loop itself**. Using [`HfA

Internally used error class for signalling an optional dependency was not found.

Context manager to capture `logging` streams Args: logger: 'logging` logger object Returns: The captured output is avai

AddedToken represents a token to be added to a Tokenizer An AddedToken can have special options defining the way it should behave.

Trainer is a simple but feature-complete training and eval loop for PyTorch, optimized for 🤗 Transformers. Args: model ([`PreTrained

A class regrouping all the information to solve a problem on which we will evaluate agents. Args: task (`str` ou `list[str]`):

Utility class containing a conversation and its history. This class is meant to be used as an input to the [`ConversationalPipeline`]. The co

Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same leng

Data collator that will dynamically pad the inputs received. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]

Implements Adam algorithm with weight decay fix as introduced in [Decoupled Weight Decay Regularization](https://arxiv.org/abs/1711.05101).

r""" Base class for all configuration classes. Handles a few parameters common to all models' configurations as well as methods for loading/do

A single training/test example for simple sequence classification. Args: guid: Unique id for the example. text_a: string. Th

Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`], [`~tokenization_utils_base.PreTrainedTokenizerBase.enc

This class can be used to create a list of [`LogitsProcessor`] or [`LogitsWarper`] to subsequently process a `scores` input tensor. This clas

Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`Pr

Holds the basic information about a new model for the add-new-model-like command. Args: model_name (`str`): The model name.

This is a wrapper class about all possible attributes and features that you can play with a model that has been loaded using `optimum` api fo

Mixin that contain utilities for preparing image features.

Evaluation output (always contains labels), to be used to compute metrics. Parameters: predictions (`np.ndarray`): Predictions of th

`Memory` NamedTuple have a single field `bytes` and you can get a human readable str of the number of mega bytes by calling `__repr__`

r""" Class that holds a configuration for a generation task. A `generate` call supports the following generation methods for text-decoder, tex

Callback that will save and push the model to the Hub regularly. By default, it pushes once per epoch, but this can be changed with the `save

r""" Holds the output of the image processor specific `__call__` methods. This class is derived from a python dictionary and can be used as a

r""" [`BeamScorer`] implementing standard beam search decoding. Adapted in part from [Facebook's XLM beam search code](https://github.com

r""" :class:`~pytorch_transformers.BertConfig` is the configuration class to store the configuration of a `BertModel`. Argum

This class can be used to create a list of [`TFLogitsProcessor`] to subsequently process a `scores` input tensor. This class inherits from li

This is a wrapper class about all possible attributes and features that you can play with a model that has been loaded using `bitsandbytes`.

Classification Head for transformer encoders

Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`Pr

r""" A special [`Constraint`] that is fulfilled by fulfilling just one of several constraints. Args: nested_token_ids (`List[List[int

r""" [`Constraint`] enforcing that an ordered sequence of tokens is included in the output. Args: token_ids (`List[int]`):

This is a wrapper class about all possible attributes and features that you can play with a model that has been loaded using `auto-awq` libra

A class responsible for properly gathering tensors (or nested list/tuple of tensors) on the CPU by chunks. If our dataset has 16 samples wit

This class can be used to stop generation whenever the full generated number of tokens exceeds `max_length`. Keep in mind for decoder-only ty

Mesh object that is used for handling certain graph operations.

Named Entity Recognition pipeline using any `ModelForTokenClassification`. See the [named entity recognition examples](../task_summary#named-

Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass Loose reference https://en.wik

Multi-turn conversational pipeline. Example: ```python >>> from transformers import pipeline, Conversation # Any model with a c

Masked language modeling prediction pipeline using any `ModelWithLMHead`. See the [masked language modeling examples](../task_summary#masked-

An error raised when the interpretor cannot evaluate a Python expression, due to syntax error or unsupported operations.

A utility attention mask class that allows one to: - Create a causal 4d mask - Create a causal 4d mask with slided window

r""" Outputs: `Tuple` comprising various elements depending on the configuration (config) and inputs: **last_hidden_state**: ``torch.Float

Same as CaptureStd but captures only stderr

Same as CaptureStd but captures only stdout

A dataset that calls prepare_seq2seq_batch.

AdaFactor pytorch implementation can be used as a drop in replacement for Adam original fairseq code: https://github.com/pytorch/fairseq/blob

The best run found by a hyperparameter search (see [`~Trainer.hyperparameter_search`]). Parameters: run_id (`str`): The

Wrapper for `contextlib.ExitStack` which enters a collection of context managers. Adaptation of `ContextManagers` in the `fastcore` library.

Fully Connected layer with on the fly adaptive mask. If needed, a score matrix is created to store the importance of each associated weight.

r""" [`LogitsProcessor`] enforcing a min-length by setting EOS probability to 0. Args: min_length (`int`): The minimum le

r""" Base class for all models. [`PreTrainedModel`] takes care of storing the configuration of the models and handles methods for loading,

container for quantization state components to work with Params4bit and similar classes

Distributed Sampler that subsamples indices sequentially, making it easier to collate all results at the end. Even though we only use this s

Constructs a SquadResult which can be used to evaluate a model's output on the SQuAD dataset. Args: unique_id: The unique identifier

Table Question Answering pipeline using a `ModelForTableQuestionAnswering`. This pipeline is only available in PyTorch. Example: ``

r""" [`LogitsWarper`] that performs top-k, i.e. restricting to the k highest probability elements. Args: top_k (`int`): T

[`LogitsWarper`] that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off. Args: top_p (`float`):

Module class that surfaces all objects but only performs associated imports when the objects are requested.

Pipeline that aims at extracting spoken text contained within some audio. The input can be either a raw waveform or a audio file. In case of

Data collator used for permutation language modeling. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses

Data collator used for language modeling that masks entire words. - collates batches of tensors, honoring their tokenizer's pad_token -

r""" [`LogitsProcessor`] that enforces no repetition of encoder input ids n-grams for the decoder ids. See [ParlAI](https://github.com/faceboo

This class can be used to create a list of [`FlaxLogitsProcessor`] or [`FlaxLogitsWarper`] to subsequently process a `scores` input tensor. T

r""" [`FlaxLogitsWarper`] for temperature (exponential scaling output probability distribution). Args: temperature (`float`):

r""" [`FlaxLogitsWarper`] that performs top-k, i.e. restricting to the k highest probability elements. Args: top_k (`int`):