Reduces a dictionary of metric lists by computing the mean of each list. Args: metrics: A dictionary mapping metric names to lists o

Reduces a dictionary of metric lists by computing the mean of each list. Args: metrics: A dictionary mapping metric names to lists o

broadcast tensor from rank0 across mp_group

broadcast tensor across mp_group

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

broadcast tensor across mp_group

broadcast tensor from rank0 across mp_group

fetch tensor

fetch tensor

fetch tensor

fetch tensor

model: AutoModelForCausalLM

Replace _flash_attention_forward to _ulysses_flash_attention_forward

Deprecated. Please use copy_to_local instead.

Create a dataset. Arguments: data_config: The data config. tokenizer (Tokenizer): The tokenizer. processor (Processor): T

Create a dataset. Arguments: data_config: The data config. tokenizer (Tokenizer): The tokenizer. processor (Processor): T

Get FSDP wrap policy for the module. Args: module: The module to get wrap policy for config: Configuration for wrap policy

Check if a path is a non-local (HDFS) path. Args: path (str): The path to check. Returns: bool: True if the path is an HDFS

Check if a path is a non-local (HDFS) path. Args: path (str): The path to check. Returns: bool: True if the path is an HDFS

Compute per-token log-probabilities for the given labels. Uses a Flash-Attention–based cross-entropy (if available) for efficient backward,

Pad a DataProto to size divisible by size_divisor Args: size_divisor (int): size divisor Returns: data: (DataProto): the pad

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Submit a chat completion request to chat scheduler and wait until it is done. To submit multiple requests in parallel, please use `generate_se

Decorator to add a timeout to a function. Args: seconds: The timeout duration in seconds. use_signals: (Deprecated) This is

broadcast tensor across mp_group

broadcast tensor across mp_group

broadcast tensor across mp_group

broadcast tensor across mp_group

Create a base TransformerConfig with common parameters across different model architectures. TODO: (ycl) use dataclass or converter config?

Create a base TransformerConfig with common parameters across different model architectures. TODO: (ycl) use dataclass or converter config?

from https://github.com/sgl-project/sglang/blob/844e2f227ab0cce6ef818a719170ce37b9eb1e1b/python/sglang/srt/utils.py#L905 Broadcast inputs from sr

Computes throughput metrics for PPO training. This function calculates performance metrics related to token processing speed, includ

Estimate the FLOPS based on the number of valid tokens in the current batch and the time taken. Args: batch_seqlens (Lis

Get the resource pool of the worker_cls

end of sentence token can be int or list: 1 or [1, 2] e.g. response_id = torch.tensor([[20, 10, 34, 1, 0, 0, 0],

end of sentence token can be int or list: 1 or [1, 2] e.g. response_id = torch.tensor([[20, 10, 34, 1, 0, 0, 0],

Puts a list of data into the Ray object store in parallel using a thread pool. Args: data_list (List[Any]): A list of Python objects

Reduces a dictionary of metric lists by computing the mean, max, or min of each list. The reduce operation is determined by the key name:

Note that this operation is in-place

Reorder the data on single controller such that each dp rank gets similar total tokens

Reorder the data on single controller such that each dp rank gets similar total tokens

broadcast tensor in tp shards across mp_group