Functions — github.com/Mwie1024/Extra-CoT

hub / github.com/Mwie1024/Extra-CoT / functions

Functions1,850 in github.com/Mwie1024/Extra-CoT

⨍ Functions 1,850◇ Types & classes 174↳ Endpoints 4

↓ 1 callersFunctionanswers_equal

(pred: str, gold: str)

src/training/qwen1p7b_rl/dataset/rl_dataset/build_diff_by_model_ratio.py:67

↓ 1 callersFunctionanswers_equal

(pred: str, gold: str)

src/training/qwen1p7b_rl/dataset/rl_dataset/build_diff.py:67

↓ 1 callersFunctionanswers_equal

(pred: Optional[str], gold: Optional[str])

src/training/qwen1p7b_rl/dataset/rl_dataset/filter_by_ratio_window.py:80

↓ 1 callersFunctionappend_jsonl

(output_path: str, obj: dict)

src/data/compressor/dataset_preparation/API/gpt-4o.py:194

↓ 1 callersFunctionappend_jsonl

(output_path: str, obj: dict)

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:197

↓ 1 callersFunctionappend_jsonl

(path: str, obj: dict, lock: threading.Lock)

src/data/compressor/dataset_preparation/API/api_result_completiness/retry_empty_chunks.py:154

↓ 1 callersMethodapply

(self, **kwargs)

llamafactory/src/llamafactory/data/formatter.py:57

↓ 1 callersMethodapply

(self, **kwargs)

llamafactory/src/llamafactory/data/formatter.py:73

↓ 1 callersFunctionapply_liger_kernel

( config: "PretrainedConfig", model_args: "ModelArguments", is_trainable: bool, require_logits

llamafactory/src/llamafactory/model/model_utils/liger_kernel.py:30

↓ 1 callersFunctionapply_ranges_to_tokens

把规范化后的 ranges（1-based 闭区间字符串）应用到 token 列表，得到“只保留”的拼接文本。

src/data/compressor/dataset_preparation/API/gpt-4o.py:325

↓ 1 callersFunctionapply_ranges_to_tokens

把规范化后的 ranges 应用到 token 列表，得到“只保留”的拼接文本。

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:360

↓ 1 callersMethodaudit_chunks

(self, original_text: str, chunk_spans: List[Tuple[int,int]])

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/gpt_chunk_camel_data.py:607

↓ 1 callersFunctionaudit_prechunked_jsonl

仅读取“已分块”的 JSONL（每行是一个样本 dict），使用其中的 chunk_spans 或按 chunks 还原 spans，运行 SmartCoTChunkerV2.audit_chunks 做质量审计。

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/audit_chunk_result.py:26

↓ 1 callersFunctionautocast_projector_dtype

r"""Cast projector output to half precision for fine-tuning quantized VLMs.

llamafactory/src/llamafactory/model/model_utils/visual.py:129

↓ 1 callersFunctionbalanced_take_per_bucket

尽量均匀；不足再二次分配。返回被选样本的全局索引列表。

src/data/sft/metamath_145k_query/select_sft_subset.py:95

↓ 1 callersMethodbatch_inference

(self, batch_input: dict[str, "torch.Tensor"])

llamafactory/src/llamafactory/eval/evaluator.py:72

↓ 1 callersFunctionbatched_forward_longformer

对一批 (question, cot) 做一次 encode+前向，返回 per-sample 元组列表： (keep_scores, offsets, prefix_len, full_text, orig_chars)

src/data/compressor_longformer/validate/longformer_pipeline/longformer_tokenizer_compression.py:249

↓ 1 callersFunctionbatched_forward_longformer

对一批 (question, cot) 做一次 encode+前向，返回 per-sample 元组列表： (keep_scores, offsets, prefix_len, full_text, orig_chars)

src/data/sft/longformer_pipeline/longformer_compressor.py:220

↓ 1 callersMethodbco_loss

( self, chosen_logps: "torch.Tensor", rejected_logps: "torch.Tensor", referenc

llamafactory/src/llamafactory/train/dpo/trainer.py:160

↓ 1 callersFunctionbuild_basis_and_example

返回 (basis_note, example_line) 两个字符串

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:171

↓ 1 callersFunctionbuild_basis_and_example

(width: int, index_basis: str)

src/data/compressor/dataset_preparation/API/api_result_completiness/retry_empty_chunks.py:135

↓ 1 callersFunctionbuild_basis_and_index_text

(question:str, chunk_text:str, min_zpad:int, index_basis:str, prompt_detailed:b

src/data/compressor/dataset_preparation/API/api_result_completiness/retry_empty_chunks.py:284

↓ 1 callersFunctionbuild_exclude_set

(path: str, key_mode: str = "q", strip_asy: bool = False, key_from: str = "query_only")

src/data/compressor_longformer/validate/datasets/sample_exlusive_1k.py:250

↓ 1 callersFunctionbuild_global_attention_mask

Build global_attention_mask for Longformer: tokens whose char offsets are within the question prefix get 1.

src/data/compressor_longformer/eval/compare_longformer_llmlingua2/compare.py:184

↓ 1 callersFunctionbuild_global_attention_mask

为 Longformer 构建 global_attention_mask：question 段（L0 < prefix_len）置 1 offsets: List[(L0,R0)]

src/data/compressor_longformer/validate/longformer_pipeline/longformer_tokenizer_compression.py:105

↓ 1 callersFunctionbuild_global_attention_mask

为 Longformer 构建 global_attention_mask：question 段（L0 < prefix_len）置 1 offsets: List[(L0,R0)]

src/data/sft/longformer_pipeline/longformer_compressor.py:76

↓ 1 callersFunctionbuild_gold_by_id

(a_path: str, a_id_keys: List[str], a_resp_keys: List[str],

src/data/compressor_longformer/validate/eval_utils/tmp.py:107

↓ 1 callersFunctionbuild_gpt4o_chunk

cap: 最大删除占比上限（如 0.55 表示 ≤55%）。设为 None 则不在提示中写删除上限。返回: system_text, user_text, aux({'tokens':..., 'math_map':...})

src/data/compressor/dataset_preparation/API/gpt_4o_format.py:40

↓ 1 callersFunctionbuild_hf_tokenizer

(llmlingua_path: str)

src/data/compressor_longformer/validate/Tokenskip_pipeline/llmlingua2_compression.py:130

↓ 1 callersFunctionbuild_id2sample

(data: List[dict])

src/data/compressor/dataset_preparation/API/api_result_completiness/retry_empty_chunks.py:187

↓ 1 callersFunctionbuild_index_view_for_chunk

返回：masked_text / tokens / offsets / index_text（1-based, zpad=3）

src/data/compressor/dataset_preparation/API/gpt-4o.py:163

↓ 1 callersFunctionbuild_index_view_for_chunk

( question: str, chunk_text: str, min_zpad: int, index_basis: str, prompt_detailed: bool )

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:382

↓ 1 callersFunctionbuild_keep_indices

(keep_spec: Dict, n: Optional[int] = None)

src/data/compressor/dataset_preparation/index_retrival.py:51

↓ 1 callersFunctionbuild_llamafactory_dataset

( original_path: str, longformer_dir: str = None, lingua_dir: str = None, ratios: List[float]

src/data/compressor_longformer/validate/build_llamafactory_input.py:195

↓ 1 callersFunctionbuild_llamafactory_dataset_special

( original_path: str, longformer_dir: str = None, lingua_dir: str = None, ratios: List[float]

src/data/compressor_longformer/validate/build_llamafactory_input_special_token.py:193

↓ 1 callersFunctionbuild_messages_qwen

(user_text: str)

src/eval/qwen1p7b_eval/vllm_eval.py:53

↓ 1 callersFunctionbuild_messages_qwen

(user_text: str)

src/eval/qwen1p7b_eval/special_token_vllm.py:89

↓ 1 callersFunctionbuild_messages_qwen

(user_text: str)

src/eval/qwen1p7b_eval/mmlu-stem/eval_vllm.py:94

↓ 1 callersFunctionbuild_mixed_dataset

( original_path: str, longformer_dir: str, lingua_dir: str, ratios: List[float],

src/data/compressor_longformer/validate/build_mixture_llamafactory.py:161

↓ 1 callersFunctionbuild_one_sample_strict

精确原位版本：不重写原文。把 ranges（基于 masked_text 的 token indexes）映射到原文字符级 keep spans，再对齐到 Longformer token。

src/data/compressor/dataset_preparation/label_training_data/label_training_data.py:156

↓ 1 callersFunctionbuild_prompt

(user_text: str, model_type: str)

src/data/compressor_longformer/validate/eval_utils/evaluate_local.py:96

↓ 1 callersFunctionbuild_prompt

(user_text: str, model_type: str)

src/data/compressor_longformer/validate/eval_utils/eval_lora.py:100

↓ 1 callersFunctionbuild_prompt_local

(user_text: str, model_type: str)

src/training/qwen1p7b_rl/dataset/vllm_inference.py:94

↓ 1 callersFunctionbuild_ratio_token_map

(buckets: List[float])

src/training/qwen1p7b_rl/dataset/rl_4k/convert_data_to_verl.py:52

↓ 1 callersFunctionbuild_ratio_token_map

(buckets: List[float])

src/training/qwen1p7b_rl/dataset/rl_tiktoken_10k/convert_data_to_verl.py:52

↓ 1 callersFunctionbuild_ratio_token_map

(buckets: List[float])

src/training/qwen1p7b_rl/dataset/rl_model_ratio_v2/convert_data_to_verl.py:58

↓ 1 callersFunctionbuild_sample_for_ratio

只做两件事： 1) input = 原始样本的 question + 对应 <COMP_xx>（1.0 -> <COMP_100>） 2) output = compressed_cot + "\\n\\nThe final answer is: $\\boxed{

src/data/compressor_longformer/validate/build_llamafactory_input_special_token.py:136

↓ 1 callersFunctionbuild_selected_record

( original: Dict[str, Any], keep_keys: List[str], derived: Dict[str, Any], fill_missing_with_n

src/data/sft/metamath_145k_query/filter_correct_answer.py:71

↓ 1 callersFunctionbuild_special_token_map

0.8 -> <COMP_80>, 1.0 -> <COMP_100>

src/data/compressor_longformer/validate/build_llamafactory_input_special_token.py:117

↓ 1 callersFunctionbuild_special_token_map

0.8 -> <comp80>, 1.0 -> <comp100>

src/data/compressor_longformer/validate/build_llamafactory_input.py:145

↓ 1 callersFunctionbuild_special_token_map

(buckets: List[float], prefix: str = "COMP_")

src/data/compressor_longformer/validate/build_mixture_llamafactory.py:109

↓ 1 callersFunctionbuild_tokenizer

(enc_name: str)

src/data/sft/longformer_pipeline/query_result/Compression/SFT_actual_ratio_intesection/filter_bad_sample.py:50

↓ 1 callersFunctionbuild_user_text

ratio_injection: - 'tokenskip'：对齐 TokenSkip（Qwen: <|eot_id|>γ<|eot_id|>；Llama3: 在末尾追加一行 γ） - 'kv' ：明文键值（compression_ratio: 0.7）

src/data/compressor_longformer/validate/eval_utils/evaluate_local.py:61

↓ 1 callersFunctionbuild_user_text

ratio_injection: - 'tokenskip'：对齐 TokenSkip（Qwen: <|eot_id|>γ<|eot_id|>；Llama3: 在末尾追加一行 γ） - 'kv' ：明文键值（compression_ratio: 0.7）

src/data/compressor_longformer/validate/eval_utils/eval_lora.py:65

↓ 1 callersFunctionbuild_user_text_special

(query: str, ratio: float, special_prefix: str = "COMP_")

src/eval/qwen1p7b_eval/vllm_eval.py:39

↓ 1 callersFunctionbuild_user_text_special

(query: str, ratio: float, special_prefix: str = "COMP_")

src/eval/qwen1p7b_eval/special_token_vllm.py:78

↓ 1 callersFunctionbuild_user_text_special

(query: str, ratio: float, special_prefix: str = "COMP_")

src/eval/qwen1p7b_eval/mmlu-stem/eval_vllm.py:78

↓ 1 callersFunctionbuild_user_text_special

(query: str, ratio: float, special_prefix: str = "COMP_")

src/training/qwen1p7b_rl/dataset/vllm_inference.py:86

↓ 1 callersFunctioncall_ranges_api

(view: Dict[str,Any], model:str, max_tokens:int, temperature:float, force_json:bool)

src/data/compressor/dataset_preparation/API/api_result_completiness/retry_empty_chunks.py:329

↓ 1 callersFunctioncall_vllm_chat

走 /v1/chat/completions；开启 logprobs 以便读取首 token

src/training/qwen1p7b_rl/dataset/rl_tiktoken_10k/validata_verl_mean_response_length.py:75

↓ 1 callersFunctioncategorize

按类别统计：去掉空白后只含 '.' 记为 dot_only，只含 ',' 记为 comma_only，二者混合记为 mixed

src/data/sft/longformer_pipeline/query_result/autofollow_sft_dataset/clear_consecutive_symbol_exclude_three_dot.py:27

↓ 1 callersFunctionchar_spans_from_topk

根据 token offset（字符级）与 keep_mask，生成字符级区间； - 仅保留 [start_char, end) 内的字符（用于跳过 <Q>...<SEP> 前缀） - 将小于等于 merge_gap_chars 的间隙合并，减少碎片

src/data/compressor_longformer/eval/preview_longformer.py:55

↓ 1 callersFunctioncheck_dependencies

r"""Check the version of the required packages.

llamafactory/src/llamafactory/extras/misc.py:95

↓ 1 callersFunctioncheck_for_duplicates

检查是否有重复的ID

src/data/compressor/dataset_preparation/camel/camel_filter/merge_camel.py:94

↓ 1 callersMethodcheck_jsonl_file

检查JSONL文件中所有样本的LaTeX完整性

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/check_latex_completiness.py:175

↓ 1 callersMethodcheck_jsonl_file

检查JSONL文件中所有样本的LaTeX完整性

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/check_latex_completiness.py:175

↓ 1 callersMethodcheck_latex_integrity

检查文本中LaTeX公式的完整性

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/check_latex_completiness.py:30

↓ 1 callersMethodcheck_latex_integrity

检查文本中LaTeX公式的完整性

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/check_latex_completiness.py:30

↓ 1 callersFunctioncheck_think_tags

检查response中是否有成对的<think>和</think>标签 Args: cot: 要检查的response文本 Returns: bool: 如果有成对的标签返回True，否则返回False

src/data/compressor/dataset_preparation/camel/camel_filter/filter_token_num_and_format.py:6

↓ 1 callersFunctioncheck_vllm_server

(base_url: str)

src/data/compressor_longformer/validate/inference_metamath.py:262

↓ 1 callersFunctioncheck_vllm_server

(base_url: str)

src/data/compressor_longformer/validate/eval_utils/evaluate.py:300

↓ 1 callersFunctioncheck_vllm_server

(base_url: str)

src/training/qwen1p7b_rl/dataset/vllm_inference.py:276

↓ 1 callersFunctionchoose_best_of

返回 (best_idx, reason, chosen_text) 规则：正确 > 有boxed > 最长

src/training/qwen1p7b_rl/dataset/vllm_inference.py:178

↓ 1 callersFunctionchoose_target_first_monotonic

(pass_map: Dict[float,bool])

src/training/qwen1p7b_rl/dataset/rl_dataset/build_diff_by_model_ratio.py:113

↓ 1 callersFunctionchoose_target_first_monotonic

(pass_map: Dict[float,bool])

src/training/qwen1p7b_rl/dataset/rl_dataset/build_diff.py:120

↓ 1 callersFunctionchoose_target_first_monotonic

返回 (chosen_r, non_mono_flag)。严格单调：选通过列表中最小 r，且对所有 r' >= r 都必须 True；否则返回 (None, True) 记为 non-mono。

src/training/qwen1p7b_rl/dataset/rl_dataset/filter_by_ratio_window.py:133

↓ 1 callersFunctionchunk_by_tokens_strict

Strict chunking: ensures each chunk, once re-encoded with add_special_tokens=True, does not exceed tokenizer.model_max_length.

src/data/compressor_longformer/eval/compare_longformer_llmlingua2/compare.py:67

↓ 1 callersFunctionchunk_by_tokens_strict

STRICT chunking: ensure each chunk, when re-encoded with add_special_tokens=True, will NOT exceed tokenizer.model_max_length.

src/data/compressor_longformer/validate/Tokenskip_pipeline/llmlingua2_compression.py:154

↓ 1 callersFunctionchunk_cot_data

对单个问题-CoT对进行分块

src/data/compressor/dataset_preparation/smart_chunker.py:117

↓ 1 callersMethodchunk_text

完整的chunking流程

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/chunk_camel_data.py:392

↓ 1 callersMethodchunk_text

完整的chunking流程

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/claude_chunk_camel_data.py:390

↓ 1 callersFunctionclean_think_regions

仅在 <think>...</think> 内： - 先统计 RUN_RE 匹配到的每个连续串 - 再把这些连续串替换为单个 '.'，**但恰好为 '...'（连续无空格的三个点）不替换** 返回 (清洗后的文本, 是否含<think>, 是否发生修改)

src/data/sft/longformer_pipeline/query_result/autofollow_sft_dataset/clear_consecutive_symbol_exclude_three_dot.py:37

↓ 1 callersMethodclose_all

(self)

src/data/compressor/dataset_preparation/API/gpt-4o.py:372

↓ 1 callersMethodclose_all

(self)

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:602

↓ 1 callersMethodclose_sub

(self, idx: int)

src/data/compressor/dataset_preparation/API/gpt-4o.py:367

↓ 1 callersMethodclose_sub

(self, id: int)

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:597

↓ 1 callersFunctioncoerce_boollike

(v: Any)

src/data/compressor_longformer/validate/Tokenskip_pipeline/llmlingua2_compression.py:87

↓ 1 callersFunctioncollect_query_set

(b_path: str, key: str, case_sensitive: bool)

src/data/sft/metamath_145k_query/filter_jsonl_by_query.py:26

↓ 1 callersFunctioncombine_question_response

从两个文件中提取对应的question和response，组合成新的数据集 Args: correct_file: 包含response的文件路径 (merged_camel_18k_correct.jsonl) filtered_file

src/data/compressor/dataset_preparation/camel/camel_filter/final_dataset.py:4

↓ 1 callersFunctioncomp_token

(ratio_str: str, zero_pad: bool)

src/data/sft/longformer_pipeline/query_result/autofollow_sft_dataset/rewrite_follow_to_unified_json_v3.py:47

↓ 1 callersFunctioncomp_token

(ratio: str, keep_zero_pad: bool)

src/data/sft/longformer_pipeline/query_result/autofollow_sft_dataset/rewrite_follow_to_unified_json_v1.py:74

↓ 1 callersFunctioncomp_token

(ratio: str, keep_zero_pad: bool)

src/data/sft/longformer_pipeline/query_result/autofollow_sft_dataset/rewrite_follow_to_unified_json_v2.py:79

↓ 1 callersFunctioncompress_from_raw_text

(raw_text: str, keep_spec: Dict, restore_math: bool = False)

src/data/compressor/dataset_preparation/index_retrival.py:144

↓ 1 callersFunctioncompress_one

(question: str, cot: str, tok, model, keep

src/data/compressor_longformer/eval/preview_longformer.py:107

↓ 1 callersFunctioncompress_output_llmlingua2

For each record: - Take FULL model_output (no <think> filtering) - Strictly chunk by tokenizer budget (including specials) - Co

src/data/compressor_longformer/validate/Tokenskip_pipeline/llmlingua2_compression.py:226

↓ 1 callersFunctioncompress_with_llmlingua_by_chunks

Strict 512-style chunking, per-chunk compression, then concatenate per ratio. Returns: ratio -> {"compressed_text", "orig_tok", "kept_tok", "

src/data/compressor_longformer/eval/compare_longformer_llmlingua2/compare.py:141

↓ 1 callersFunctioncompress_with_longformer

Compress the same COT text at multiple ratios (no chunking).

src/data/compressor_longformer/eval/compare_longformer_llmlingua2/compare.py:310

↓ 1 callersFunctioncompress_with_longformer

返回： ratio -> list[compressed_items]，每个 items 的字段与 LLMLingua2 输出一致

src/data/compressor_longformer/validate/longformer_pipeline/longformer_tokenizer_compression.py:320

↓ 1 callersFunctioncompress_with_longformer

返回： ratio -> list[compressed_items]，每个 items 的字段与 LLMLingua2 输出一致

src/data/sft/longformer_pipeline/longformer_compressor.py:291

↓ 1 callersFunctioncompute_avg_tokens

(jsonl_path: str, field_path: str, tokenizer_pref: Optional[str], print_each: bool)

src/data/compressor_longformer/validate/eval_utils/count_token.py:77

← previous next →601–700 of 1,850, ranked by callers