A DataProto is a data structure that aims to provide a standard protocol for data exchange between functions. It contains a batch (TensorDict

An LLM for generating texts from given prompts and sampling parameters. This class includes a tokenizer, a language model (possibly distributed

We assume the dataset contains a column that contains prompts and other information

Configuration for the KV cache. Args: block_size: Size of a cache block in number of tokens. gpu_memory_utilization: Fraction of

Model loader that will set model weights to random values.

Model loader that will set model weights to random values.

Model loader that will set model weights to random values.

Configuration for the distributed execution. Args: pipeline_parallel_size: Number of pipeline parallel groups. tensor_parallel_si

Scheduler configuration. Args: max_num_batched_tokens: Maximum number of tokens to be processed in a single iteration.

DataProtoFuture aims to eliminate actual data fetching on driver. By doing so, the driver doesn't have to wait for data so that asynchronous

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

The reward manager.

This is an in-memory SFTDataset

Adaptive KL controller described in the paper: https://arxiv.org/pdf/1909.08593.pdf

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

Model loader that can load the model weights from partitioned megatron model.

Arguments for vLLM engine.

Arguments for vLLM engine.

Arguments for vLLM engine.

Disable All Megatron Timing with FakeTimers

Fixed KL controller.

Model loader that can load the model weights from model's full params.

Model loader that can load the model weights from model's full params.

Model loader that can load the model weights from model's full params.

LlamaRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla

LlamaRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendev

download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_format: The for

download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_for

download_dir: Directory to download and load the weights, default to the default cache directory of huggingface. load_for

A logger that writes data to a local file and optionally prints to console.

A memory buffer is a contiguous torch tensor that may combine multiple tensors sharing with the underlying memory. It must have a unique type

Configuration for the model. Args: model: Name or path of the huggingface model to use. tokenizer: Name or path of the huggingfac

Configuration for the model. Args: model: Name or path of the huggingface model to use. tokenizer: Name or path of the huggingfac

Configuration for the model. Args: model: Name or path of the huggingface model to use. tokenizer: Name or path of the huggingfac

Configuration for optimizer.

Multi-headed attention from 'Attention Is All You Need' paper