broadcast tensor across mp_group

broadcast tensor in tp shards across mp_group

TODO: optimize this. - We can use async ops - We can use only one allgather Args: tensors: size: group:

We compute dense reward here so that we can directly train RL without SFT

Get the resource pool of the worker_cls

Make an iterator from the DataProto. This is built upon that TensorDict can be used as a normal Pytorch dataset. See https://pytorch.org/tenso

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Rotates half the hidden dims of the input.

Set ulysses sequence parallel process group.

broadcast tensor in tp shards across mp_group

Build the memory buffer given weight_buffer_meta Args: weight_buffer_meta: contains mapping from name to a dictionary containing shape an

for cuda memory alignment, make sure alignment by 128-bits

Compute logits given a batch of data. Args: data (DataProto): a batch of data represented by DataProto. It must contain key ```in

The training loop of PPO. The driver process only need to call the compute functions of the worker group through RPC to construct the

Generate sequences

Returns an array of tuples, containing the string `input`, sympy `output` using the provided sympy `func`, and `symbolically` boolean for cal

Apply ceil() then return the ceil-ed expression. expr: Expr - sympy expression as an argument to ceil()

Apply floor() then return the floored expression. expr: Expr - sympy expression as an argument to floor()

Init resource pool and worker group