Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the current Data

If distributed is initialized, print only on rank 0.

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: chunks (int): the number of

Copy src from hdfs to local if src is on hdfs or directly return src. If cache_dir is None, we will use the default cache dir of the system. Note

Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` Args: batch_keys (list, optional): a list of strings indicati

Concat a list of DataProto. The batch is concatenated among dim=0. The meta_info is assumed to be identical and will use the first one.

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

Repeat the batch data a specified number of times. Args: repeat_times (int): Number of times to repeat the data.

Select a subset of the DataProto via batch_keys and meta_info_keys Args: batch_keys (list, optional): a list of strings indicatin

Get ulysses sequence parallel world size.

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Args: text (`str`, `List[str]` or

Create a DataProto from a dict of tensors. This assumes that 1. All the tensor in tensors have the same dim0 2. Only dim0 is the batch

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Get ulysses sequence parallel process group.

Applies the SFT template to conversation. An example of conversation: conversation = [ { "role":

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

Append a new message.

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

Compute mean of tensor with a masked values.

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_seq_l

Union with another DataProto. Union batch and meta_info separately. Throw an error if - there are conflict keys in batch and they are

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

end of sentence token can be int or list: 1 or [1, 2] e.g. eos_token=1 response_id: [0, 0, 2, 42, 3, 5, 1, 0, 0] eos_mask: [1, 1,

Create a huggingface processor to process multimodal data. Args: name_or_path (str): The name of the processor. Returns: tra

Register a new conversation template.

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Args: pil_img (PIL.Image): [H, W, 3] in PIL.Image in RGB Returns: x (np.ndarray): [3, self.image_size, self

broadcast tensor across mp_group

broadcast tensor across mp_group

Replace _flash_attention_forward to _ulysses_flash_attention_forward

TODO: optimize this. Technically, we only need one broadcast

Get a conversation template.

Get FSDP wrap policy for the module. Args: module: The module to get wrap policy for config: Configuration for wrap policy

Build the model.

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group