Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the current Data

If distributed is initialized, print only on rank 0.

Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` Args: batch_keys (list, optional): a list of strings indicati

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: chunks (int): the number of

Copy src from hdfs to local if src is on hdfs or directly return src. If cache_dir is None, we will use the default cache dir of the system. Note

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

Create a DataProto from a dict of tensors. This assumes that 1. All the tensor in tensors have the same dim0 2. Only dim0 is the batch

Concat a list of DataProto. The batch is concatenated among dim=0. The meta_info is assumed to be identical and will use the first one.

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

Select a subset of the DataProto via batch_keys and meta_info_keys Args: batch_keys (list, optional): a list of strings indicatin

Repeat the batch data a specified number of times. Args: repeat_times (int): Number of times to repeat the data.

Get ulysses sequence parallel world size.

Get ulysses sequence parallel process group.

Compute mean of tensor with a masked values.

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_seq_l

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

This method forwards all its arguments to Qwen2TokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring o

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Args: text (`str`, `List[str]` or

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

Union with another DataProto. Union batch and meta_info separately. Throw an error if - there are conflict keys in batch and they are

注入 JS 自动滚动左列容器到底部

TODO: optimize this. Technically, we only need one broadcast

Create attention mask from input ids.

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

end of sentence token can be int or list: 1 or [1, 2] e.g. eos_token=1 response_id: [0, 0, 2, 42, 3, 5, 1, 0, 0] eos_mask: [1, 1,

Build the model.

Split the batch into a list of micro_batches, where the max_token_len is smaller than max_token_len and the number of valid tokens in each micro b

刷新 Memory Graph 可视化

broadcast tensor across mp_group

broadcast tensor across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

内部方法，用于保存索引和元数据。

TODO: optimize this. - We can use async ops - We can use only one allgather Args: tensors: size: group:

Concatenate tensors and handle padding.

Estimate the FLOPS based on the number of valid tokens in the current batch and the time taken. Args: batch_seqlens (Lis