Compute strict chunk core length so that: len(ids_with_specials) <= model_max_length.

Compute STRICT core length so that len(ids)+specials <= model_max_length (e.g., 512).

r"""Clean the cached memory and resets the runner.

r"""Post-process the response text. Based on: https://huggingface.co/spaces/Lyte/DeepSeek-R1-Distill-Qwen-1.5B-Demo-GGUF/blob/main/app.py

r"""Return the corresponding dataset processor.

r"""Return the default logging level.

r"""Get gemma3 token type ids for computing loss. Returns: batch_token_type_ids: shape (batch_size, seq_length)

r"""Return the ollama template.

r"""Get paligemma token type ids for computing loss. It is slightly different with the original token type ids where the prompt part is 0. R

r"""Prepare the tokenized dataset to perform AutoGPTQ. Do not use tensor output for JSON serialization.

r"""Load a single dataset and aligns it to the standard format.

r"""Make nested list of images.

r"""Helper function to read JSON/JSONL files using fsspec.

按顺序在原文中搜索每个 chunk 的首次出现位置（从上一个 chunk 的末尾继续找）， 以尽量还原 (start, end) 坐标。若出现找不到（例如重复片段、预处理有空格差异等），返回 None。

r"""Abort the processes recursively in a bottom-up way.

r"""Add empty thought to assistant message.

r"""Set module as a leaf module to skip partitioning in deepspeed zero3.

r"""Align the dataset to a specific format. Aligned dataset: _prompt: [{"role": "user", "content": "..."}] * (2T - 1) _response: [{"role"

根据策略返回用于构造 key 的题干候选列表。 key_from: - 'query_only' : 只用 query 系列字段（与您的验重逻辑一致） - 'messages_only': 只用 messages[0].content - '

reduce SUM over ranks, return python float

分析chunk分布情况

分析JSONL文件中每个样本的token分布情况

分析JSONL文件中每个样本的token分布情况

分析数据集中question+response的token长度分布 Args: input_file: 输入的jsonl文件路径 output_prefix: 输出文件前缀（可选，用于保存图片）