Create test DataProto for compute_log_prob method

Execute a method on a single worker remotely. Args: worker: The worker actor handle method_name: Name of the method t

fetch tensor in tp shards

fetch tensor in tp shards

Get current memory usage. Note that CPU device memory info is always 0. Args: unit (str, optional): The unit of memory measurement.

Handle the generating state: generate model response and check for tool calls.

Handle the interacting state: get user input from interaction.

Handle the processing tools state: execute tool calls and prepare tool responses.

Check if a node has a docstring.

Log rollout data to disk. Args: log_rollout_meta (BatchMeta): The batch_meta of rollout data reward_extra_in

Process the padded outputs from _run_agent_loop and combine them into a batch.

Process tokens for a single message or a group of messages. Args: messages: List of message dictionaries sta

Process proprioceptive features and append to vision features

Query the collect info for a given mesh name. Args: mesh_name (str): Name of the mesh to query collect info for.

Query the dispatch info for a given mesh name. Args: mesh_name (str): Name of the mesh to query dispatch info for

Slice response from unpad model output. Args: tensor: model output tensor of shape [bsz, 1] data: TensorDict with "prompt_ids", "

TODO: optimize this. Technically, we only need one broadcast

for cuda memory alignment, make sure alignment by 128-bits

Calculate the workload for a dense transformer block based on sequence length. FLOPs = 12 * hidden_size^2 * seqlen + 2 * hidden_size * seqlen

LLM chat interface. Args: messages: Inputted messages. functions: Inputted functions for function calling. OpenAI for

Adapted from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py Args: token_level_rewards: `(torch.Tensor)`

Compute the clipped policy objective and related metrics for PPO. Adapted from https://github.com/huggingface/trl/blob/main/trl/trainer/

We compute dense reward here so that we can directly train RL without SFT

Compute reward score for model solutions with robust handling of various formats. Returns a weighted combination of: - Accuracy reward (

Create a mock configuration with multiple interactions.

Create a TensorDict from a dict of tensors and non_tensors. Note that this requires tensordict version at least 0.10

Context manager entry for switching the engine and model into evaluation mode. Usage: with engine.eval_mode():

Execute a method on all workers asynchronously. Args: method_name: Name of the method to execute *args: Positional ar

Create a worker group from existing detached workers. Args: name_prefix: Prefix for worker names worker_names: Names

Generate sequences for a batch of prompts. Args: batch (DataProto): Input batch. Returns: DataProto: Output

Tracker file rescords the latest chckpoint during training to restart from.

Dynamically import and return the interaction class.

Get the forward function for given model architecture.

Get the weight converter for given model architecture.

Get the last learning rate from the optimizer parameter scheduler.

Get the optimizer parameter scheduler for Megatron.

Returns the number of input images for the vision backbone. Returns: Number of images expected in the input

Returns the number of vision patches output by the vision backbone. Returns: Number of patches per image

Get the policy loss with a given name. Args: name: `(str)` The name of the policy loss. Returns: `(callable)`: T

end of sentence token can be int or list: 1 or [1, 2] e.g. response_id = torch.tensor([[20, 10, 34, 1, 0, 0, 0],

Gets the position ids for Qwen2-VL, it should be generated before sharding the sequence. The batch dim has been removed and the input_ids sho