Handle the pending state: prepare the prompt and start generation.

Log validation data

Normalize answer expressions.

Start profiling for all worker groups if profiling is enabled.

Stop profiling for all worker groups if profiling is enabled.

Inner function that handles the core timing logic. Args: name (str): The name/identifier for this timing measurement. timing_raw

Update the input_ids, attention_mask, position_ids, and loss_mask of the request in additive manner.

Async generate method that mirrors AsyncEngine.async_generate interface. This method provides compatibility with AsyncEngine's async_generate

Performs bootstrap resampling to estimate statistics of metrics. This function uses bootstrap resampling to estimate the mean and standard d

from https://github.com/sgl-project/sglang/blob/844e2f227ab0cce6ef818a719170ce37b9eb1e1b/python/sglang/srt/utils.py#L905 Broadcast inputs from sr

A helper method to check if the passed module's key name matches any of the exclude modules in the adapter_config. Adapted from https://githu

A helper method to check if the passed module's key name matches any of the target modules in the adapter_config. Adapted from https://github

Compute rollout correction weights and apply rejection sampling. Computes importance sampling weights to correct for off-policy issues between

Create a dataset.

Executes an API search query, caching by API type.

Flush the cache of the server. This method repeatedly attempts to flush the server cache until successful. The flush operation will n

Create a DataProto from a TensorDict. This assumes that 1. All the tensor in tensor_dict have the same dim0 2. Only dim0 is the batch

torch.nn.utils.clip_grad_norm_ cann't run on cpu parameter DTensor

Calculate the pipeline sharding configuration for Megatron-LM. Args: layer_num: Total number of layers in the model. pp_size: Num

Get the generation prompt ids for rollout engine. Because rollout engine(SGLang) requires the ids to be a list, we need to convert t

Get the resource pool of the worker_cls

Choose the dimension that can divide fsdp_size to avoid padding

Function that gets visible devices keyword name. Returns: 'CUDA_VISIBLE_DEVICES' or `ASCEND_RT_VISIBLE_DEVICES`

Compute KL divergence given logprob and ref_logprob. Optionally using straight through to bind k2 on other kl penalty compute method for unbiased

Convert TensorDict from left-right padding to no-padding format. Args: data: TensorDict with "input_ids", "attention_mask", "respons

Calculate and log metrics related to sequence length imbalance before and after partitioning. Args: seqlen_list (List[int]): A list

Args: messages: List[dict] List of dictionaries with keys 'role' and 'content' Example: messages = [{'role': 'use

All gather across tp group to make each rank has identical input.

Reduces a dictionary of metric lists by computing the mean of each list. Args: metrics: A dictionary mapping metric names to lists o

Reduce timing information across all processes. This function uses distributed communication to gather and sum the timing information from al

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Set ulysses sequence parallel process group.

Sleep all rollout replica instances.

Move FSDP model and/or optimizer to CPU or GPU with offload support.

Calculate topk items take-up ratio, and min/max timing across all ranks.

Update the weights of the rollout model. Args: weights: A generator that yields the name of the weight tensor and the tensor itse

Binds worker methods to the WorkerGroup based on registered attributes. Args: user_defined_cls (type): The class containing metho

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group