Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Move model parameters, optimizer states, or both to the specified device. Args: device: Target device identifier.

Update method for fixed KL controller (no-op). Args: current_kl (float): Current KL divergence value (unused). n_step

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: split_size (int): the size

Return the corresponding torch attribute based on the device type string. Returns: module: The corresponding torch device namespace, or to

Return current device id based on the device type. Returns: device index

Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` Args: batch_keys (list, optional): a list of strings indicati

Log GPU memory usage information. Args: head (str): A descriptive header for the memory usage log message. logger (logging.Logger

If distributed is initialized, print only on rank 0.

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: chunks (int): the number of

Context manager for timing with platform markers. This utility function measures the execution time of code within its context, accumulates t

_summary_ Log a message with rank information using a logger. This function logs the message only if `log_only_rank_0` is False or if the rank

Function that gets the torch.device based on the current machine. This currently only supports CPU, CUDA, NPU. Returns: device

Copy files/directories from HDFS to local cache with validation. Args: src (str): Source path - HDFS path (hdfs://...) or local filesyste

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

Select a subset of the DataProto via batch_keys and meta_info_keys Args: batch_keys (list, optional): a list of strings indicatin

Convert an OmegaConf DictConfig to a dataclass. Args: config: The OmegaConf DictConfig or dict to convert. dataclass_type: T

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_seq_l

Concat a list of DataProto. The batch is concatenated among dim=0. The meta_info is assumed to be identical and will use the first one.

Repeat the batch data a specified number of times. Args: repeat_times (int): Number of times to repeat the data.

Create a tool instance. Args: instance_id: The instance id of the tool. Returns: The instance id of the tool

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

Create a DataProto from a dict of tensors. This assumes that 1. All the tensor in tensors have the same dim0 2. Only dim0 is the batch

[Deprecated] convert config Args: hf_config (PretrainedConfig): _description_ megatron_config (_type_): _description_ Return

Generate sequences

Register the dp_rank for a given mesh name. This function is meant to be called by the worker Args: mesh_name (str):

Aggregate the loss matrix into a scalar. Args: loss_mat: `(torch.Tensor)`: shape: (bs, response_length) loss_mas

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Args: text (`str`, `List[str]` or

Create a huggingface processor to process multimodal data. Args: name_or_path (str): The name of the processor. Returns: tra

Instantiate or load the model, optimizer, and learning rate scheduler. Should prepare all components necessary for training or evalu

In megatron, the model and optimizer storage are: - bf16 parameter data chunked in model parallel group - fp32 grad chunked in model para

Append values from new_data to lists in data. For each key in new_data, this function appends the corresponding value to a list stored under

Get the resource pool of the worker_cls

Get ulysses sequence parallel world size.

Context manager for basic timing without NVTX markers. This utility function measures the execution time of code within its context and accum

_summary_ Thread-safe directory creation function that ensures the directory is created even if multiple processes attempt to create it simult

Get ulysses sequence parallel process group.

Preprocess packed sequences CP splits sequence into CP*2 chunks, and each GPU gets 2 chunks (GPU0 gets first and last chunks, GPU1 gets s

Split a batch into micro-batches by total token count, with optional DP sync and padding. Args: batch (TensorDict): must include "at

Select specific indices from the DataProto. Args: idxs (torch.Tensor or numpy.ndarray or list): Indices to select

Calculate global rank with support for CP/EP parallelism

More aggressive GPU memory cleanup function, tries to release PyTorch reserved but unallocated memory. Args: force_sync: Whether to

Return nccl backend type based on the device type. Returns: nccl backend type string.

Sleep engine to offload model weights and discard kv cache.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the current Data

broadcast tensor from rank0 across mp_group

Compute the attention mask for the response part of the sequence. This function extracts the portion of the attention mask that corresponds to th

Get the rank of this worker in the distributed setup.

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Create a DataProto from a dict of tensors and non_tensors

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

Build the inverse of an index mapping. Args: idx_map (Sequence[int]): Sequence where idx_map[i] = j. Returns: List[int]

Check if a path is a non-local (HDFS) path. Args: path (str): The path to check. Returns: bool: True if the path is an HDFS

Load a external data type based on the file path and type name

Compute the mean of `values` over elements selected by `mask`. Args: values (Tensor): Input tensor. mask (Tensor): Boolean o

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

broadcast tensor from rank0 across mp_group

broadcast tensor across mp_group

broadcast tensor from rank0 across mp_group

Dump rollout/validation samples as JSONL.

fetch tensor

fetch tensor

Create a base TransformerConfig with common parameters across different model architectures. TODO: (ycl) use dataclass or converter config?