Add a request to the engine's request pool. The request is added to the request pool and will be processed by the scheduler as `engin

allgather params of all pp ranks. Return a list of handles

This function should return a class instance that delegates the calls to every cls in cls_dict

Implement flush method for compatibility.

Args: input_ids: input ids. shape (batch_size, seq_length) attention_mask: attention_mask. shape (batch_size, seq_le

We assume: - The model takes input: (input_ids, attention_mask, position_ids). No rmpad for the input - The communication sha

Get all the parameters of the models in all pp ranks Returns: params: List[List[Dict[str, Tensor]]]: a list of parameters in all

e.g. end of sentence token=1 response_id: [0, 0, 2, 42, 3, 5, 1, 0, 0] eos_mask: [1, 1, 1, 1, 1, 1, 1, 0, 0]

Return world size for the tensor model parallel group.

Return a dictionary containing name to a shape and dtype.

The answer will be considered correct if: (a) it normalizes to the same string as the ground truth answer OR (b) sympy can simplify t

Return the result of gcd() or lcm(), as UnevaluatedExpr f: str - name of function ("gcd" or "lcm") args: List[Expr] - list of function a

NOTE: This method is a hack from the open-sourced version without asertion of world_size = tp * pp Initialize model parallel groups.

NOTE: This method is a hack from the open-sourced version without asertion of world_size = tp * pp Initialize model parallel groups.

load all model params to cuda

Log data to file and optionally console. Args: data: Dictionary containing metrics/data to log step:

Compute the log_probs from logits with rmpad input_ids and logits. Note that logits_rmpad = model(input_ids_rmpad). For each sentences, there is a

Make minibatch iterator for updating the actor Args: data (DataProto): a DataProto containing keys ``input_ids``

r"""Works like os.makedirs() but supports hdfs. Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except tha

Exact match of math if and only if: 1. numerical equal: both can convert to float and are equal 2. symbolic equal: both can convert to sy