Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the current Data

If distributed is initialized, print only on rank 0.

Create a DataProto from a dict of tensors. This assumes that 1. All the tensor in tensors have the same dim0 2. Only dim0 is the batch

Copy src from hdfs to local if src is on hdfs or directly return src. If cache_dir is None, we will use the default cache dir of the system. Note

Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` Args: batch_keys (list, optional): a list of strings indicati

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: chunks (int): the number of

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

Concat a list of DataProto. The batch is concatenated among dim=0. The meta_info is assumed to be identical and will use the first one.

Union with another DataProto. Union batch and meta_info separately. Throw an error if - there are conflict keys in batch and they are

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

Select a subset of the DataProto via batch_keys and meta_info_keys Args: batch_keys (list, optional): a list of strings indicatin

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_seq_l

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

Repeat the batch data a specified number of times. Args: repeat_times (int): Number of times to repeat the data.

Get ulysses sequence parallel world size.

Compute mean of tensor with a masked values.

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

Get ulysses sequence parallel process group.

Generate sequences

Split the batch into a list of micro_batches, where the max_token_len is smaller than max_token_len and the number of valid tokens in each micro b

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

Deprecated. Please use copy_to_local instead.

Create a random mask given input_ids. Support left padding and right padding. Process: - Sample valid token length - Sample left_padding l

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Build the model.

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

TODO: optimize this. Technically, we only need one broadcast

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

end of sentence token can be int or list: 1 or [1, 2] e.g. eos_token=1 response_id: [0, 0, 2, 42, 3, 5, 1, 0, 0] eos_mask: [1, 1,

Get FSDP wrap policy for the module. Args: module: The module to get wrap policy for config: Configuration for wrap policy

Init resource pool and worker group

broadcast tensor across mp_group

broadcast tensor across mp_group

broadcast tensor in tp shards across mp_group