Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the current Data

If distributed is initialized, print only on rank 0.

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: chunks (int): the number of

Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` Args: batch_keys (list, optional): a list of strings indicati

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Concat a list of DataProto. The batch is concatenated among dim=0. The meta_info is assumed to be identical and will use the first one.

Copy files/directories from HDFS to local cache with validation. Args: src (str): Source path - HDFS path (hdfs://...) or local filesyste

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

Select a subset of the DataProto via batch_keys and meta_info_keys Args: batch_keys (list, optional): a list of strings indicatin

Create a DataProto from a dict of tensors. This assumes that 1. All the tensor in tensors have the same dim0 2. Only dim0 is the batch

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_seq_l

Repeat the batch data a specified number of times. Args: repeat_times (int): Number of times to repeat the data.

Union with another DataProto. Union batch and meta_info separately. Throw an error if - there are conflict keys in batch and they are

In megatron, the model and optimizer storage are: - bf16 parameter data chunked in model parallel group - fp32 grad chunked in model para

Aggregate the loss matrix into a scalar. Args: loss_mat: `(torch.Tensor)` shape: (bs, response_length) loss_mask:

Create a tool instance. Args: instance_id: The instance id of the tool. Returns: The instance id of the tool

Get ulysses sequence parallel process group.

Get ulysses sequence parallel world size.

Sleep engine to offload model weights and discard kv cache.

Calculate global rank with support for CP/EP parallelism

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Args: text (`str`, `List[str]` or

Wake up engine to load model weights and build kv cache.

broadcast tensor from rank0 across mp_group

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

broadcast tensor from rank0 across mp_group

broadcast tensor across mp_group

broadcast tensor from rank0 across mp_group

fetch tensor

fetch tensor

Returns: entropy: # (bs, response_len) log_probs: # (bs, response_len) top_k_log_probs: # (bs, response_l

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

Build the model.

end of sentence token can be int or list: 1 or [1, 2] e.g. response_id = torch.tensor([[20, 10, 34, 1, 0, 0, 0],

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

Compute mean of tensor with a masked values.

Split a batch into micro-batches by total token count, with optional DP sync and padding. Args: batch (TensorDict): must include "at

This is the equivalent of torch.repeat_interleave(x, dim=2, repeats=n_rep). The hidden states go from (batch, seqlen, num_key_value_heads, he

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

broadcast tensor across mp_group

broadcast tensor across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

Create a base TransformerConfig with common parameters across different model architectures. TODO: (ycl) use dataclass or converter config?