broadcast tensor in tp shards across mp_group

Build multimodal embeddings and attention mask

Convert observation to model input format

Args: log_probs: (batch_size, traj_len, ...) entropy: (batch_size, traj_len, ...) mask: (batch_siz

Generates the completions for the input prompts. NOTE: This class automatically batches the given prompts, considering the memory con

Args: end_step: (batch_size,), traj_len: Returns: mask: (batch_size, traj_len),

Obtain the grasp pose through the marked grasp point. - actor: The instance of the object to be grasped. - arm_tag: The arm t

获取 v2 在 v1 上的投影向量

Get the resource pool of the worker_cls

Get the tensor model parallel group the caller rank belongs to.

Get the tensor model parallel group the caller rank belongs to.

Get the tensor model parallel group the caller rank belongs to.

Get the tensor model parallel group the caller rank belongs to.

Return world size for the tensor model parallel group.

Make an iterator from the DataProto. This is built upon that TensorDict can be used as a normal Pytorch dataset. See https://pytorch.org/tenso

Rotates half the hidden dims of the input.

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

Union with another DataProto. Union batch and meta_info separately. Throw an error if - there are conflict keys in batch and they are

Update the policy with an iterator of DataProto Args: data (DataProto): an iterator over the DataProto that returns by

broadcast tensor in tp shards across mp_group

Get and verify the model's maximum length.

Process proprioceptive features and append to vision features

Run diffusion-based action prediction

TODO: optimize this. - We can use async ops - We can use only one allgather Args: tensors: size: group:

Build the memory buffer given weight_buffer_meta Args: weight_buffer_meta: contains mapping from name to a dictionary containing shape an

for cuda memory alignment, make sure alignment by 128-bits

r"""Works like shutil.copy() for file, and shutil.copytree for dir, and supports hdfs. Copy data and mode bits ("cp src dst"). Return the file's

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokeniz

Post-Process Observation for robotwin 2.0

Generate sequences

Get the dimensionality of the policy's action space.

获取从 v1 到 v2 的旋转矩阵

e.g. end of sentence token=1 response_id: [0, 0, 2, 42, 3, 5, 1, 0, 0] eos_mask: [1, 1, 1, 1, 1, 1, 1, 0, 0]

Returns the number of input images for the vision backbone. Returns: Number of images expected in the input

Returns the number of vision patches output by the vision backbone. Returns: Number of patches per image