A DataProto is a data structure that aims to provide a standard protocol for data exchange between functions. It contains a batch (TensorDict

Sharding manager to support data resharding when using FSDP + Ulysses

This is an in-memory SFTDataset

Configuration for the KV cache. Args: block_size: Size of a cache block in number of tokens. gpu_memory_utilization: Fraction of

Model loader that will set model weights to random values.

Model loader that will set model weights to random values.

Model loader that will set model weights to random values.

An LLM for generating texts from given prompts and sampling parameters. This class includes a tokenizer, a language model (possibly distributed

Configuration for the distributed execution. Args: pipeline_parallel_size: Number of pipeline parallel groups. tensor_parallel_si

We assume the dataset contains a column that contains prompts and other information

Note that this trainer runs on the driver process on a single CPU/GPU node.

Define a resource pool specification. Resource pool will be initialized first. Mapping

Scheduler configuration. Args: max_num_batched_tokens: Maximum number of tokens to be processed in a single iteration.

DataProtoFuture aims to eliminate actual data fetching on driver. By doing so, the driver doesn't have to wait for data so that asynchronous

The implementation of a simple digit completion task. The prompt is a sequence of numbers with fixed difference. The task is to complete the

Used to count mfu during training loop Example: flops_counter = FlopsCounter(config) flops_achieved, flops_promised = flops_

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

The reward manager.

Adaptive KL controller described in the paper: https://arxiv.org/pdf/1909.08593.pdf

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

Arguments for vLLM engine.

Arguments for vLLM engine.

Arguments for vLLM engine.

Disable All Megatron Timing with FakeTimers

Fixed KL controller.

Model loader that can load the model weights from model's full params.

Model loader that can load the model weights from model's full params.

Model loader that can load the model weights from model's full params.

LlamaRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla

LlamaRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev

download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_format: The for

download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_for

download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_for

A memory buffer is a contiguous torch tensor that may combine multiple tensors sharing with the underlying memory. It must have a unique type

Note that we do not design MemoryBufferModuleWrapper as an nn.Module due to - It will change the checkpoint name

Configuration for the model. Args: model: Name or path of the huggingface model to use. tokenizer: Name or path of the huggingfac

Configuration for the model. Args: model: Name or path of the huggingface model to use. tokenizer: Name or path of the huggingfac

Configuration for the model. Args: model: Name or path of the huggingface model to use. tokenizer: Name or path of the huggingfac

MegatronWorkerGroup will query each worker of its megatron rank info and store it inside the WorkerGroup so that the dispatcher can use it to

Configuration for optimizer.

Multi-headed attention from 'Attention Is All You Need' paper

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config: LlamaConfig

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config: LlamaConfig

Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] This model definition supports pip

A group of tokenizers that can be used for LoRA adapters.

A group of tokenizers that can be used for LoRA adapters.

A group of tokenizers that can be used for LoRA adapters.

A group of tokenizers that can be used for LoRA adapters.

A worker class that executes (a partition of) the model on a GPU. Each worker is associated with a single GPU. The worker is responsible for

A worker class that executes (a partition of) the model on a GPU. Each worker is associated with a single GPU. The worker is responsible for