A wrapper class for Ray actors with initialization arguments. This class extends ClassWithInitArgs to provide additional functionality for co

The response from a tool execution.

A group of Ray workers that can be managed collectively. This class extends WorkerGroup to provide Ray-specific functionality for creating an

A DataProto is a data structure that aims to provide a standard protocol for data exchange between functions. It contains a batch (TensorDict

HTTP-based adapter for SGLang engines. This adapter allows interaction with SGLang engines through HTTP requests instead of direct engine cal

Worker profiler config. The inheritance from BaseConfig provides omegaconf.DictConfig-like interface for a dataclass config. Args: d

Async token bucket for rate limiting with variable token consumption. The token bucket algorithm is a classic rate limiting technique that allows

NPU profiler. Initialized in a worker to control the NPU profiler.

Configuration for critic model training. The inheritance from BaseConfig provides omegaconf.DictConfig-like interface for a dataclass config.

FSDP optimizer configuration extending base OptimizerConfig. Args: optimizer (str): Optimizer class name (e.g., "AdamW", "AdamW8bit", "_A

A unified tracking interface for logging experiment data to multiple backends. This class provides a centralized way to log experiment metrics, p

Manage FSDP checkpointing in SPMD training. - Saves/loads per-rank sharded model & optimizer states - Persists full lr_scheduler and RNG

Used to count mfu during training loop Example: flops_counter = FlopsCounter(config) flops_achieved, flops_promised = flops_

Base optimizer configuration. Args: lr (float): learning rate. Must be specified. lr_warmup_steps_ratio (float): Warmup steps rat

Reward loop manager with rate limiting for API-based reward functions. This manager implements a sophisticated three-layer rate limiting system

NPU profiler too; config.

Load and preprocess RLHF data from Parquet files. - Caches files locally. - Reads into a HuggingFace Dataset and tokenizes prompts.

A dispatcher that delegates to specific profilers based on config.tool. Supported tools: - nsys: NsightSystemsProfiler - npu: NPUProfiler

Sharding manager to support data resharding when using FSDP + Ulysses

Asynchronous HTTP-based adapter for SGLang engines. This class inherits from HttpServerAdapter and adds async capabilities for non-blocking H

Configuration for FSDP (Fully Sharded Data Parallel). The inheritance from BaseConfig provides omegaconf.DictConfig-like interface for a dataclas

Configuration for Megatron parallelism. The inheritance from BaseConfig provides omegaconf.DictConfig-like interface for a dataclass config.

RolloutSkip skips sequence generation during rollout by attempting to load previously dumped data. If no dumped data is found, it generates n

Configuration for actor model training. The inheritance from BaseConfig provides omegaconf.DictConfig-like interface for a dataclass config.

Configuration for FSDP-based critic model training. The inheritance from CriticConfig provides all base critic configuration plus FSDP-specific s

Define a resource pool specification. Resource pool will be initialized first.

Agent loop output.

FSDP DataParallel PPO Actor or Ref worker Args: config (ActorConfig): Actor config actor_module (nn.Module): Actor or ref module

The schema of a tool in OpenAI format.

Configuration for Mcore module wrapper.

Mcore optimizer configuration extending base OptimizerConfig. Args: optimizer (str): Optimizer name; default is "adam". lr (float

Define a resource pool specification. Resource pool will be initialized first.

Reward model manager.

A distributed worker that handles initialization and configuration for distributed training. This class manages worker initialization, configurat

Encapsulates all state variables for the agent loop.

Configuration for the algorithm. The inheritance from BaseConfig provides omegaconf.DictConfig-like interface for a dataclass config. Args:

The data model for async rollout.

A demo interaction for calculating the reward of gsm8k. - `start_interaction`: start a interaction instance for a trajectory. - `generate_res

Checkpoint manager for Megatron-LM distributed training. This class manages the saving and loading of model checkpoints in a Megatron-LM

Dataset for multi-turn conversations where each assistant response should be trained

The schema of parameters in OpenAI format.

The knowledge data format output from the retrieval

This is an in-memory SFTDataset Arguments: config (OmegaConf): the data config

This is a special tool for data storage

This worker can be instantiated as a standalone actor or a standalone rollout or a standalone reference policy or a hybrid engine based on th

This worker can be instantiated as a standalone actor or a standalone reference policy or a hybrid engine based on the config.rollout

Agent loop manager that manages a group of agent loop workers.

This is a widely applicable agent integrated with RAG capabilities and function call ability.

A class to manage multiple OpenAI compatible LLM servers. This class provides - Load balance: least requests load balancing - Sticky sess

DataProtoFuture aims to eliminate actual data fetching on driver. By doing so, the driver doesn't have to wait for data so that asynchronous

Configuration for FSDP actor models. The inheritance from BaseConfig provides omegaconf.DictConfig-like interface for a dataclass config. Ar

Configuration for Megatron-based critic model training. The inheritance from CriticConfig provides all base critic configuration plus Megatron-sp

Configuration for model merger operations. Args: operation (str): Operation type - 'merge' or 'test'. backend (str): Backend type

The schema of a parameter in OpenAI format.

The schema of a function in OpenAI format.

A PyTorch profiler wrapper class for collecting performance metrics. TODO(haibin.lin): this should implement the DistProfiler interface, and the

Distributed PPO trainer using Ray for scalable reinforcement learning. This trainer orchestrates distributed PPO training across multiple nodes a

Define a resource pool specification. Resource pool will be initialized first. Mapping