A class that keeps all conversation history.

Processes an answer similar to Eval AI copied from https://github.com/facebookresearch/mmf/blob/c46b3b3391275b4181567db80943473a8

r""" Constructs an image processor. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image

An indexable dataset based on a list of shards. The dataset is either given as a list of shards with optional options and name, or as a URL p

This class is implemented by Jason Lu.

Initialization. Arguments: local_attention (Module): local attention with q,k,v sequence_process_group (ProcessGroup): sequence p

Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`GemmaDecoderLayer`] Args: config: GemmaConfig

Initialization. Arguments: ulysses_pg (ProcessGroup): ulysses process group ring_pg (ProcessGroup): ring process group sc

A class that accesses samples in a tar file. The tar file must follow WebDataset conventions. The tar file is indexed when the IndexedTarSamples

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config: LlamaConfig

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MistralDecoderLayer`] Args: config: MistralCo

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MixtralDecoderLayer`] Args: config: MixtralCo

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MixtralDecoderLayer`] Args: config: MixtralCo

Base class for causal language model (or autoregressive) with mixture of experts outputs. Args: loss (`torch.FloatTensor` of shape `

Base class for model's outputs, with potential hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `

Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`SiglipEncoderLayer`]. Args:

Base class for outputs of question answering models using a [`~modeling_utils.SQuADHead`]. Args: loss (`torch.FloatTensor` of shape

Fake file-like stream object that redirects writes to a logger instance.

Fake file-like stream object that redirects writes to a logger instance.

Initialization. Arguments: local_attention (Module): local attention with q,k,v sequence_process_group (ProcessGroup): sequence p

r""" The ValueHead class implements a head for GPT2 that returns a scalar for each output token.

Adaptive KL controller described in the paper: https://arxiv.org/pdf/1909.08593.pdf

A [`TrainerCallback`] that handles autoresume. Args: interval: interval (in number of iterations) between checks as to w

A sampler that samples in chunks and then shuffles the samples within each chunk. This preserves locality of reference while still shuffling the

Iterable dataset that returns constant length chunks of tokens from stream of text files. The dataset also formats the text before tokenizati

Output class for the diffusers pipeline to be finetuned with the DDPO trainer Args: images (`torch.Tensor`): The generat

Output class for the diffusers scheduler to be finetuned with the DDPO trainer Args: latents (`torch.Tensor`): Predicted

r""" DPO DataCollator class that pads the tokenized inputs to the maximum length of the batch. Args: pad_token_id (`int` defaults to 0

Dataset for supervised fine-tuning.

Collate examples for supervised fine-tuning. This class is originally implemented by the LLaVA team and modified by Haotian Tang.

Collate examples for supervised fine-tuning. This class is originally implemented by the LLaVA team and modified by Haotian Tang.

A simple non-blocking exclusive lock using fcntl.

Fixed KL controller.

Implement the scaled dot product attention with softmax. Arguments --------- softmax_scale: The temperature to use for the softmax att

Multi-headed attention from 'Attention Is All You Need' paper

Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InternEncoderLayer`]. Args:

A class that manages a cache of shards. The cache is a LRU cache that stores the local names of the shards as keys and the downloaded paths as

LlamaRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla

Llama flash attention module. This module inherits from `LlamaAttention` as the weights of the module stays untouched. The only required chan

LlamaRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev

This handle assumes that no need to copy data out from a contiguous tensor

Wrapper to change the behavior for parameter sharding

Wrapper to change the behavior for parameter sharding