Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the current Data

If distributed is initialized, print only on rank 0.

Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` Args: batch_keys (list, optional): a list of strings indicati

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: chunks (int): the number of

Create a DataProto from a dict of tensors. This assumes that 1. All the tensor in tensors have the same dim0 2. Only dim0 is the batch

Copy src from hdfs to local if src is on hdfs or directly return src. If cache_dir is None, we will use the default cache dir of the system. Note

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Sleep engine to offload model weights and discard kv cache.

Union with another DataProto. Union batch and meta_info separately. Throw an error if - there are conflict keys in batch and they are

Concat a list of DataProto. The batch is concatenated among dim=0. The meta_info is assumed to be identical and will use the first one.

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_seq_l

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

Create a tool instance. Args: instance_id: The instance id of the tool. Returns: The instance id of the tool

Select a subset of the DataProto via batch_keys and meta_info_keys Args: batch_keys (list, optional): a list of strings indicatin

Repeat the batch data a specified number of times. Args: repeat_times (int): Number of times to repeat the data.

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

r"""Works like shutil.copy() for file, and shutil.copytree for dir, and supports hdfs. Copy data and mode bits ("cp src dst"). Return the file's

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Args: text (`str`, `List[str]` or

Generate sequences

r"""Works like os.makedirs() but supports hdfs. Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except tha

In megatron, the model and optimizer storage are: - bf16 parameter data chunked in model parallel group - fp32 grad chunked in model para

Get ulysses sequence parallel world size.

Aggregate the loss matrix into a scalar. Args: loss_mat: `(torch.Tensor)` shape: (bs, response_length) loss_mask:

Build the model.

Get ulysses sequence parallel process group.

Compute mean of tensor with a masked values.

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

Wake up engine to load model weights and build kv cache.

Calculate global rank with support for CP/EP parallelism

Create a DataProto from a dict of tensors and non_tensors

Load a external data type based on the file path and type name

Split the batch into a list of micro_batches, where the max_token_len is smaller than max_token_len and the number of valid tokens in each micro b

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

Replace _flash_attention_forward to _ulysses_flash_attention_forward

Note that this operation is in-place

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

fetch tensor

fetch tensor

Deprecated. Please use copy_to_local instead.

Create a random mask given input_ids. Support left padding and right padding. Process: - Sample valid token length - Sample left_padding l

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

Create a huggingface processor to process multimodal data. Args: name_or_path (str): The name of the processor. Returns: tra

Init resource pool and worker group

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

This is the equivalent of torch.repeat_interleave(x, dim=2, repeats=n_rep). The hidden states go from (batch, seqlen, num_key_value_heads, he

Submit a chat completion request to chat scheduler and wait until it is done. To submit multiple requests in parallel, please use `generate_se