r"""Works like os.makedirs() but supports hdfs. Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except tha

r"""Pre-process token ids after tokenization for VLMs.

Read data from a JSONL (JSON Lines) file. Args: file_path: Path to the JSONL file Returns: list: List of pa

r"""Regularize images to avoid error. Including reading and pre-processing.

估算文本的token数量

Get ulysses sequence parallel process group.

Pad and slice input_ids to be divisible by sp_size Pad position_ids to be divisible by sp_size. Note both input_ids_rmpad and position_i

r"""Add elements to manager.

r"""Convert slots to jinja template.

Clear all messages

Calculate tokens for a text string

Format messages for LLM by converting them to OpenAI message format. Args: messages: List of messages that can be either

Generate sequences

Split the batch into a list of micro_batches, where the max_token_len is smaller than max_token_len and the number of valid tokens in each micro b

r"""Check template. Args: model_id: the model id on hugging face hub. template_name: the template name. prompt_str: the s

given TP,DP,PP rank to get the global rank.

given TP,DP,PP rank to get the global rank.

r"""Regularizes videos to avoid error. Including reading, resizing and converting.

Replace _flash_attention_forward to _ulysses_flash_attention_forward

Factory method to create and initialize an environment

r"""Get the number of available devices.

r"""Load user config if exists.

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

Exact match of math if and only if: 1. numerical equal: both can convert to float and are equal 2. symbolic equal: both can convert to sy

r"""Save model checkpoint. Subclass and override to inject custom behavior.

r"""Collect the device memory.

broadcast tensor from rank0 across mp_group

broadcast tensor from rank0 across mp_group

r"""Optionally concatenate media path to media dir when loading from local disk.

Get friendly display name for agent

Create a random mask given input_ids. Support left padding and right padding. Process: - Sample valid token length - Sample left_padding l

Estimates pass@k of each problem and returns them in an array.

格式化字节数为可读格式

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

r"""Get the current available device.

Build the model.

获取最近的交易日（跳过周末） A股交易日规则： - 周一到周五是交易日 - 周六周日是休市 Args: date_format: 返回日期格式，默认为"%Y-%m-%d" Returns:

r"""Return stop token ids.

r"""Compute the real sequence length after truncation by the cutoff_len.

r"""Cast a torch tensor or a numpy array to a numpy array.

r"""Plot loss curves and saves the image.

Agent调用的主方法。根据核心概念查找并返回对应的few-shot示例块。 Args: core_concept (str): 需要查找的核心概念，例如 "保障相关时间"。 Returns: st

Save nested dictionary/list configuration to a Python file. Args: nested_dict_list: Configuration data structure to save

Add a message to the agent's memory. Args: role: The role of the message sender (user, system, assistant, tool). cont