Get ulysses sequence parallel world size.

Aggregate the loss matrix into a scalar. Args: loss_mat: `(torch.Tensor)`: shape: (bs, response_length) loss_mas

Append values from new_data to lists in data. For each key in new_data, this function appends the corresponding value to a list stored under

Append values from new_data to lists in data. For each key in new_data, this function appends the corresponding value to a list stored under

In megatron, the model and optimizer storage are: - bf16 parameter data chunked in model parallel group - fp32 grad chunked in model para

In megatron, the model and optimizer storage are: - bf16 parameter data chunked in model parallel group - fp32 grad chunked in model para

Gather a tensor across a process group and optionally unpad its padded elements. Args: x (Tensor): Input tensor to gather. g

Build the inverse of an index mapping. Args: idx_map (Sequence[int]): Sequence where idx_map[i] = j. Returns: List[int]

Get ulysses sequence parallel world size.

Run a command on a remote host. No conda activation; default env is assumed.

Aggregate the loss matrix into a scalar. Args: loss_mat: `(torch.Tensor)`: shape: (bs, response_length) loss_mas

Get ulysses sequence parallel process group.

Get ulysses sequence parallel process group.

Compute the mean of `values` over elements selected by `mask`. Args: values (Tensor): Input tensor. mask (Tensor): Boolean o

Spawn to a dictionary of worker groups, each with a subset of method with prefix. Args: prefix_set: Set of prefixes to create wor

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

Wake up engine to load model weights and build kv cache.

Build the inverse of an index mapping. Args: idx_map (Sequence[int]): Sequence where idx_map[i] = j. Returns: List[int]

Wake up engine to load model weights and build kv cache.

Calculate global rank with support for CP/EP parallelism

Calculate global rank with support for CP/EP parallelism

Gather a tensor across a process group and optionally unpad its padded elements. Args: x (Tensor): Input tensor to gather. g

Build the model.

Load a external data type based on the file path and type name

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

Replace _flash_attention_forward to _ulysses_flash_attention_forward

Create a DataProto from a dict of tensors and non_tensors

Create a DataProto from a dict of tensors and non_tensors

Build the model.

Create a huggingface processor to process multimodal data. Args: name_or_path (str): The name of the processor. Returns: tra

Create a huggingface processor to process multimodal data. Args: name_or_path (str): The name of the processor. Returns: tra

Get the rank of this worker in the distributed setup.

Get the rank of this worker in the distributed setup.

Reduces a dictionary of metric lists by computing the mean, max, or min of each list. The reduce operation is determined by the key name:

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Deprecated. Please use copy_to_local instead.

Create a random mask given input_ids. Support left padding and right padding. Process: - Sample valid token length - Sample left_padding l

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

Function that gets the torch.device based on the current machine. This currently only supports CPU, CUDA, NPU. Returns: device

Function that gets the torch.device based on the current machine. This currently only supports CPU, CUDA, NPU. Returns: device

Get FSDP wrap policy for the module. Args: module: The module to get wrap policy for config: Configuration for wrap policy

Return the tool's information in a standardized format for tool registration. Returns: Dict[str, Any]: Tool info

Return the tool's information in a standardized format for tool registration. Returns: Dict[str, Any]: Tool info

Compute per-token log-probabilities for the given labels. Uses a Flash-Attention–based cross-entropy (if available) for efficient backward,

Creates a batch generator suitable for Megatron pipeline parallelism, handling virtual pipeline parallelism (VPP). If VPP is used (vpp_s

Compute the mean of `values` over elements selected by `mask`. Args: values (Tensor): Input tensor. mask (Tensor): Boolean o