Shared CLI arguments for vLLM engine.

Build the memory references. The memory buffers are built using the build_memory_buffer API. This API will allocate a weight buffer pointer to the

Check the consistency of the DataProto. Mainly for batch and non_tensor_batch We expose this function as a public one so that user can call th

Only collect the data from the tp=0 and pp=last and every dp ranks

treat pp as dp. Only collect data on tp=0

Computing reward given input_ids. The transformers should output a tensor with shape [batch_size, sequence_length], and the value at [EOS]

Args: model_name: actor_override_config_kwargs: Returns:

name: name of the parameter param: training parameters infer_params (List[torch.Tensor]): a list of parameters all-gathered f

User passes in dp data. The data is dispatched to all tp/pp ranks with the same dp

treat pp as dp.

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Args: text (`str`, `List[str]` or

Helper to initialize model parallel groups if they are not initialized, or ensure tensor-parallel and pipeline-parallel sizes are equal to expecte

Helper to initialize model parallel groups if they are not initialized, or ensure tensor-parallel and pipeline-parallel sizes are equal to expecte

Helper to initialize model parallel groups if they are not initialized, or ensure tensor-parallel and pipeline-parallel sizes are equal to expecte

Safely evaluate the arithmetic equation using eval() with precautions.

Extract the equation from the solution string.

Fold a batch dim from [bsz, xxx] into [new_bsz, bsz // new_bsz, xxx]

We assume: - The model takes input: (input_ids, attention_mask, position_ids). No rmpad for the input - The communication sha

Creates an LLM engine from the engine arguments.

Creates an LLM engine from the engine arguments.

Creates an LLM engine from the engine arguments.

Creates an LLM engine from the engine arguments.

Gather the label from data. The value in label should be [0, vocab_size) Args: data: (..., vocab_size) label (torch.IntTensor) :

any score <0.4 is ok, since +- is easy the model have

Generate ground truth response given a prompt.

Return a local temp path that joins cache_dir and basename of hdfs_path Args: hdfs_path: cache_dir: Returns:

Get a model loader based on the load format.

Get a model loader based on the load format.

Get a model loader based on the load format.