Get the rank of this worker in the distributed setup.

Sleep all rollout replica instances.

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

Batch generate sequences in sync mode. Args: prompts: The input prompts. Returns: The output sequences.

Get job by ID.

Return the list of RouterReplay instances corresponding to the current micro-batch and local (pp_rank, vp_stage) layer range.

Build the inverse of an index mapping. Args: idx_map (Sequence[int]): Sequence where idx_map[i] = j. Returns: List[int]

Load an object from a module path. Args: module_path (str): See :func:`load_module`. object_name (str): The name of t

Compute the mean of `values` over elements selected by `mask`. Args: values (Tensor): Input tensor. mask (Tensor): Boolean o

broadcast tensor from rank0 across mp_group

broadcast tensor across mp_group

broadcast tensor from rank0 across mp_group

fetch tensor

fetch tensor

Create a base TransformerConfig with common parameters across different model architectures. TODO: (ycl) use dataclass or converter config?

Get output file path for a dataset type. Note: Categorized files (solved/learnable/unsolved) are saved by save_categorized() using th

Make an async HTTP request with retry logic and consistent error handling. Args: endpoint (str): The API endpoint to call (withou

Make a HTTP request with retry logic and consistent error handling. Args: endpoint (str): The API endpoint to call (without leadi

Add current progress to buffer Args: idxs: The index of processed sample added after the last save.

Manually release vLLM model and free GPU memory. Call this method when you're done with the model to release GPU resources.

Convert Hydra configs and other special types to regular Python types.

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Args: text (`str`, `List[str]` or

Estimate the FLOPS based on the number of valid tokens in the current batch and the time taken. Args: batch_seqlens (Lis

Evaluate a post-trained model using DeepEval. \b Example: sdg eval configs/eval.yaml sdg eval configs/eval.yaml --model ./che

Extract answer from LLM response. Supports two extraction modes: 1. XML-style tags (e.g., "<answer>"): Extracts content betw

Mark job as failed.

Check if a path is a non-local (HDFS) path. Args: path (str): The path to check. Returns: bool: True if the path is an HDFS

Applies a monkey patch to the forward method of a given model class to enable Ulysses sequence parallelism input slicing.

Postprocess packed sequences

Resume rollout weights or kv cache in GPU memory. Args: tags: weights or kv_cache.

Run agent loop to interact with LLM server and environment. Args: sampling_params (Dict[str, Any]): LLM sampling params.

Enable or disable expandable segments for cuda. Args: enable (bool): Whether to enable expandable segments. Used to avoid OOM.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the current Data

Convert this DataProto to TensorDict. Note that this requires tensordict version at least 0.10 Returns:

Translate all samples in a dataset to the target language. Args: dataset: Dataset to translate target_lang:

broadcast tensor across mp_group

broadcast tensor across mp_group

model: AutoModelForCausalLM

Apply monkey patch to the models for ulysses sequence parallel and fused kernel. In the end of this function forward function of the model i

Check and disable incompatible configurations for older Megatron version. Args: original_config (dict): The original model configura

r"""Works like shutil.copy() for file, and shutil.copytree for dir, and supports hdfs. Copy data and mode bits ("cp src dst"). Return the file's

Fallback implementation for older transformers versions. Returns False to disable features that require this function.

Loads the full state dict (could be only on rank 0) into the sharded model. This is done by broadcasting the parameters from rank 0 to all ot

Generate responses from VLM model with image inputs. Args: prompts: Single prompt string or list of prompts

Get FSDP wrap policy for the module. Args: module: The module to get wrap policy for config: Configuration for wrap policy

Get the resource pool of the worker_cls

Convert TensorDict from left-right padding to no-padding format. Args: data: TensorDict with "input_ids", "attention_mask", "respons

Load weights for mcore GPT model.

Save checkpoint using FSDPCheckpointManager with improved tracking

Set callback to be called after estimate_usage.

Context manager for basic timing without NVTX markers. This utility function measures the execution time of code within its context and accum

Move model parameters, optimizer states, or both to the specified device. Note that this function executes irrespective of offload co

Move FSDP model and/or optimizer to CPU or GPU with offload support. Note that this function executes irrespective of offload config.