Get the tensor model parallel group the caller rank belongs to.

Return world size for the tensor model parallel group.

Make an iterator from the DataProto. This is built upon that TensorDict can be used as a normal Pytorch dataset. See https://pytorch.org/tenso

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Rotates half the hidden dims of the input.

Set ulysses sequence parallel process group.

broadcast tensor in tp shards across mp_group

Get and verify the model's maximum length.

Build the memory buffer given weight_buffer_meta Args: weight_buffer_meta: contains mapping from name to a dictionary containing shape an

for cuda memory alignment, make sure alignment by 128-bits

The training loop of PPO. The driver process only need to call the compute functions of the worker group through RPC to construct the

Gets the number of unfinished requests.

Return world size for the tensor model parallel group.

Return world size for the tensor model parallel group.

Returns True if there are unfinished requests.

Init resource pool and worker group

NOTE: This method is a hack from the open-sourced version without asertion of world_size = tp * pp Initialize model parallel groups.

Pad a DataProto to size divisible by size_divisor Args: size_divisor (int): size divisor Returns: data: (DataProto): the pad

Note that this operation is in-place

Set input tensor to be used instead of forward()'s input. When doing pipeline parallelism the input from the previous stage comes fro

Union two tensordicts.

TODO(zhangchi.usc1992): We may change the implementation later

Returns: entropy: # (bs, response_len) log_probs: # (bs, response_len)

Get Stats to be Logged to Prometheus.

Add a request to the engine's request pool. The request is added to the request pool and will be processed by the scheduler as `engin

allgather params of all pp ranks. Return a list of handles