Copy files/directories from HDFS to local cache with validation. Args: src (str): Source path - HDFS path (hdfs://...) or local filesyste

Type text via ADB Keyboard (supports CJK and Latin characters). Requires ADB Keyboard to be installed on the device.

Union with another DataProto. Union batch and meta_info separately. Throw an error if - there are conflict keys in batch and they are

Loads all episodes from disk.

Function that gets the torch.device based on the current machine. This currently only supports CPU, CUDA, NPU. Returns: device

_summary_ Log a message with rank information using a logger. This function logs the message only if `log_only_rank_0` is False or if the rank

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

Pad a DataProto to size divisible by size_divisor Args: size_divisor (int): size divisor Returns: data: (DataProto): the pad

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_seq_l

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, head_d

Create a DataProto from a dict of tensors and non_tensors

Generate sequences