Functions — github.com/Mwie1024/Extra-CoT

hub / github.com/Mwie1024/Extra-CoT / functions

Functions1,850 in github.com/Mwie1024/Extra-CoT

⨍ Functions 1,850◇ Types & classes 174↳ Endpoints 4

↓ 2 callersFunctioniter_records_any

兼容 JSONL（每行对象/数组）和 JSON 数组文件；逐条产出 dict

src/data/compressor_longformer/validate/datasets/sample_exlusive_1k.py:34

↓ 2 callersFunctionleft_truncate_by_tokens

(text: str, tokenizer, max_tokens: int)

src/training/qwen1p7b_rl/dataset/rl_tiktoken_10k/validata_verl_mean_response_length.py:41

↓ 2 callersFunctionload_args

r"""Load the training configuration from config path.

llamafactory/src/llamafactory/webui/common.py:154

↓ 2 callersFunctionload_dir_as_index

(d: str)

src/training/qwen1p7b_rl/dataset/merge_data.py:114

↓ 2 callersFunctionload_done_indices_from_jsonl

(output_path: str)

src/data/compressor/dataset_preparation/API/gpt-4o.py:200

↓ 2 callersFunctionload_done_indices_from_jsonl

(output_path: str)

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:231

↓ 2 callersFunctionload_input_records

优先按 JSON 解析，失败则退回 JSONL。统一返回 list[dict]。

src/data/compressor/dataset_preparation/API/gpt4o_ranges_labeler.py:219

↓ 2 callersFunctionload_json

(file, encoding='utf-8')

src/data/compressor_longformer/validate/build_llamafactory_input_numeric.py:6

↓ 2 callersFunctionload_json_array

(path: Path)

src/data/sft/longformer_pipeline/query_result/autofollow_sft_dataset/merge_data_and_add_newline_after_think.py:14

↓ 2 callersFunctionload_json_or_jsonl

(path: str)

src/eval/qwen1p7b_eval/mmlu-stem/recheck_acc.py:245

↓ 2 callersFunctionload_jsonl

(file, encoding='utf-8')

src/data/compressor_longformer/validate/build_llamafactory_input_special_token.py:15

↓ 2 callersFunctionload_jsonl

(file, encoding='utf-8')

src/data/compressor_longformer/validate/build_llamafactory_input.py:15

↓ 2 callersFunctionload_jsonl

(file, encoding='utf-8')

src/data/compressor_longformer/validate/build_mixture_llamafactory.py:18

↓ 2 callersFunctionload_jsonl

(path: Path)

src/data/compressor_longformer/validate/longformer_pipeline/llama3.2_3b_instruct/real_ratio/intersect_by_question.py:45

↓ 2 callersFunctionload_jsonl

(path: Path)

src/data/sft/longformer_pipeline/intersect_by_question.py:45

↓ 2 callersFunctionload_jsonl

Yield JSON objects, skipping malformed lines.

src/data/sft/longformer_pipeline/query_result/Compression/RL_actual_ratio/sample_recat_10k_auto_data.py:69

↓ 2 callersFunctionload_jsonl

读取 JSONL 文件

src/data/sft/longformer_pipeline/query_result/Compression/SFT_actual_ratio_intesection/sample_jsonl_by_query.py:10

↓ 2 callersFunctionload_ratio_file

(dir_path: str, ratio: float)

src/data/compressor_longformer/validate/build_llamafactory_input_special_token.py:75

↓ 2 callersFunctionload_ratio_file

(dir_path: str, ratio: float)

src/data/compressor_longformer/validate/build_llamafactory_input.py:75

↓ 2 callersFunctionload_ratio_file

(dir_path: str, ratio: float)

src/data/compressor_longformer/validate/build_mixture_llamafactory.py:74

↓ 2 callersFunctionload_valuehead_params

r"""Load value head parameters from Hugging Face Hub or local disk. Returns: dict with keys `v_head.summary.weight` and `v_head.summary.bias`.

llamafactory/src/llamafactory/model/model_utils/valuehead.py:33

↓ 2 callersFunctionload_xxx_set

读取 B/C.jsonl，提取所有 xxx（按上面的模板），做 normalize 后去重成集合。忽略解析失败或无匹配的行。

src/data/sft/metamath_145k_query/diff_by_query_against_rl_train_test.py:64

↓ 2 callersFunctionmake_input_A

(question: str, comp_tok: str)

src/data/compressor_longformer/validate/build_mixture_llamafactory.py:153

↓ 2 callersFunctionmake_input_text

- numeric: Qwen => 在末尾注入 <|eot_id|>{ratio:.1f}<|eot_id|>（ratio<1.0 时注入；1.0 不注入） Llama3 => 追加一行 "compression_ratio: {ratio:.1f}"（rat

src/data/compressor_longformer/validate/build_llamafactory_input.py:158

↓ 2 callersFunctionmake_keys

返回该样本的一组去重键（列表）。 mode: - 'q' : 按题干（question/query）去重 - 'qo' : 按题干+输出去重 - 'id' : 按 id 去重（无 id 则回退到 'q'） key_from: 参见

src/data/compressor_longformer/validate/datasets/sample_exlusive_1k.py:156

↓ 2 callersFunctionmake_out_path

(in_path: Path, out_dir: Path | None, suffix: str)

src/data/compressor_longformer/validate/longformer_pipeline/llama3.2_3b_instruct/real_ratio/intersect_by_question.py:66

↓ 2 callersFunctionmake_out_path

(in_path: Path, out_dir: Path | None, suffix: str)

src/data/sft/longformer_pipeline/intersect_by_question.py:66

↓ 2 callersFunctionmaybe_add_specials

(tokenizer, tokens: List[str])

src/data/compressor_longformer/validate/eval_utils/evaluate_local.py:118

↓ 2 callersFunctionmaybe_add_specials

为 tokenizer 增加缺失的 special tokens，返回新增数量。

src/data/compressor_longformer/validate/eval_utils/eval_lora.py:122

↓ 2 callersMethodmerge_short_sentences

合并过短的句子片段

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/chunk_camel_data.py:130

↓ 2 callersMethodmerge_short_sentences

合并过短的句子片段

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/claude_chunk_camel_data.py:128

↓ 2 callersFunctionmerge_spans

(spans: List[Tuple[int,int]])

src/data/compressor/dataset_preparation/label_training_data/label_training_data.py:104

↓ 2 callersFunctionnested_detach

r"""Detach `tensors` (even if it's a nested list/tuple/dict of tensors).

llamafactory/src/llamafactory/train/trainer_utils.py:679

↓ 2 callersFunctionnorm

(x: str)

src/data/compressor_longformer/validate/eval_utils/tmp.py:102

↓ 2 callersFunctionnormalize

对比前的标准化：统一换行、压缩空白、去首尾空白。

src/data/sft/metamath_145k_query/diff_by_query_against_rl_train_test.py:32

↓ 2 callersFunctionnormalize

标准化字符串用于匹配

src/data/sft/longformer_pipeline/query_result/Compression/SFT_actual_ratio_intesection/sample_jsonl_by_query.py:19

↓ 2 callersFunctionnormalize_id

(v: Any, case_insensitive: bool = False)

src/data/compressor_longformer/validate/eval_utils/tmp.py:39

↓ 2 callersFunctionnormalize_text

(s: str)

src/data/compressor_longformer/validate/longformer_pipeline/llama3.2_3b_instruct/real_ratio/intersect_by_question.py:35

↓ 2 callersFunctionnormalize_text

(s: str)

src/data/sft/longformer_pipeline/intersect_by_question.py:35

↓ 2 callersFunctionparse_number

(s: str)

src/data/compressor_longformer/validate/eval_utils/tmp.py:77

↓ 2 callersFunctionparse_ratio_from_extra

从 extra_info 尽量还原 ratio： 1) chosen_comp_token（<COMP_40>） 2) gt_ratio / ratio_bucket 找不到则默认 2.0（表示 AUTO）

src/training/qwen1p7b_rl/dataset/rl_tiktoken_10k/validata_verl_mean_response_length.py:50

↓ 2 callersFunctionparse_ratio_probs

"1.0:0.1,0.8:0.2,0.6:0.3,0.4:0.2,0.2:0.2"

src/data/compressor_longformer/validate/build_mixture_llamafactory.py:357

↓ 2 callersFunctionpatch_valuehead_model

()

llamafactory/src/llamafactory/train/test_utils.py:113

↓ 2 callersFunctionpick_answer

(rec: Dict[str, Any])

src/data/compressor_longformer/validate/build_llamafactory_input_special_token.py:59

↓ 2 callersFunctionpick_answer

(rec: Dict[str, Any])

src/data/compressor_longformer/validate/build_mixture_llamafactory.py:59

↓ 2 callersFunctionprepare_4d_attention_mask

r"""Expand 2d attention mask to 4d attention mask. Expand the attention mask with indices from (batch_size, seq_len) to (batch_size, 1, seq_len,

llamafactory/src/llamafactory/data/collator.py:41

↓ 2 callersFunctionprocess_item

(item)

llamafactory/data/convert.py:10

↓ 2 callersFunctionread_args

r"""Get arguments from the command line or a config file.

llamafactory/src/llamafactory/hparams/parser.py:57

↓ 2 callersFunctionread_jsonl

(path:str)

src/data/compressor/dataset_preparation/API/api_result_completiness/retry_empty_chunks.py:160

↓ 2 callersFunctionread_jsonl

(path:str)

src/data/compressor/dataset_preparation/API/api_result_completiness/merge_results.py:6

↓ 2 callersFunctionrecheck_file

返回 (总数, 修正为 True 的样本数, 预测有变更数, 最终正确数)

src/eval/qwen1p7b_eval/recheck_accuracy.py:297

↓ 2 callersFunctionrecheck_file

返回 (总数, 修正为 True 的样本数, 预测有变更数, 最终正确数, 用content修正为True数)

src/eval/qwen1p7b_eval/mmlu-stem/recheck_acc.py:323

↓ 2 callersFunctionreduce_counts

(c: Dict[str, int])

src/data/compressor_longformer/train/train_longformer.py:214

↓ 2 callersFunctionrepeat_fraction

(seq: List[str], n: int)

src/data/sft/longformer_pipeline/query_result/Compression/RL_actual_ratio/filter_bad_sample.py:126

↓ 2 callersFunctionrepetition_ratio

(grams)

src/data/sft/longformer_pipeline/query_result/Compression/SFT_actual_ratio_intesection/filter_bad_sample.py:26

↓ 2 callersFunctionreplace_model

r"""Replace the default/reward modules in the model. The model is already unwrapped.

llamafactory/src/llamafactory/train/ppo/ppo_utils.py:43

↓ 2 callersFunctions

(x: str)

src/data/compressor_longformer/validate/eval_utils/evaluate.py:135

↓ 2 callersFunctionsave_args

r"""Save the training configuration to config path.

llamafactory/src/llamafactory/webui/common.py:163

↓ 2 callersFunctionsave_jsonl

(items, path)

src/data/sft/longformer_pipeline/query_result/Compression/SFT_actual_ratio_intesection/filter_bad_sample.py:78

↓ 2 callersFunctionsave_jsonl

(path: str, rows: List[Dict[str,Any]])

src/training/qwen1p7b_rl/dataset/rl_4k/convert_data_to_verl.py:80

↓ 2 callersFunctionsave_jsonl

(path: str, rows: List[Dict[str,Any]])

src/training/qwen1p7b_rl/dataset/rl_tiktoken_10k/convert_data_to_verl.py:80

↓ 2 callersFunctionsave_jsonl

(path: str, rows: List[Dict[str,Any]])

src/training/qwen1p7b_rl/dataset/rl_model_ratio_v2/convert_data_to_verl.py:86

↓ 2 callersFunctionsave_jsonl_line_safe

(obj: Dict[str, Any], out_path: str, lock_path: str)

src/data/compressor_longformer/validate/eval_utils/evaluate.py:212

↓ 2 callersFunctionsave_result_to_file

(result: Dict[str, Any], output_file: str, lock_file: str)

src/data/compressor_longformer/validate/inference_metamath.py:190

↓ 2 callersFunctionscan_tokens

(text: str, ignore_spans: Optional[List[Tuple[int,int]]] = None)

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/math_tokenizer_no_space.py:89

↓ 2 callersFunctionsetup_fs

r"""Set up a filesystem object based on the path protocol.

llamafactory/src/llamafactory/data/data_utils.py:145

↓ 2 callersFunctionsmooth

r"""EMA implementation according to TensorBoard.

llamafactory/src/llamafactory/extras/ploting.py:34

↓ 2 callersFunctionspan_ok_range

(a,b,toks_local)

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/math_tokenizer_no_space.py:507

↓ 2 callersMethodsplit_into_sentences

将文本按句子分割，但避开数学公式内的标点符号

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/chunk_camel_data.py:51

↓ 2 callersMethodsplit_into_sentences

将文本按句子分割，但避开数学公式内的标点符号

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/claude_chunk_camel_data.py:49

↓ 2 callersFunctionsympy_equal

(ea, eb)

src/eval/qwen1p7b_eval/recheck_accuracy.py:162

↓ 2 callersFunctionsympy_equal

(ea, eb)

src/eval/qwen1p7b_eval/mmlu-stem/recheck_acc.py:147

↓ 2 callersFunctiontok_count

(encoder, s: str)

src/data/sft/longformer_pipeline/query_result/Compression/RL_actual_ratio/filter_bad_sample.py:179

↓ 2 callersFunctiontokens_if_comma_list

若答案是逗号分隔的多项（如多个根/集合），返回分割后的 token 列表（去空白的原始片段）；否则返回 None。

src/training/qwen1p7b_rl/dataset/rl_dataset/check_basemodel_accuracy.py:153

↓ 2 callersFunctiontokens_len

(tok, s: str)

src/data/compressor_longformer/validate/Tokenskip_pipeline/llmlingua2_compression.py:134

↓ 2 callersFunctionwhitespace_tokens

(s: str)

src/data/compressor/dataset_preparation/API/dataset_split/sampler.py:25

↓ 1 callersMethod__init__

(self, config: "LlavaConfig")

llamafactory/src/llamafactory/model/model_utils/visual.py:85

↓ 1 callersMethod__init__

(self, manager: "Manager", demo_mode: bool = False, lazy_init: bool = True)

llamafactory/src/llamafactory/webui/chatter.py:81

↓ 1 callersMethod__init__

r"""Init a runner.

llamafactory/src/llamafactory/webui/runner.py:57

↓ 1 callersMethod__init__

(self, items: List[Dict[str, Any]])

src/data/compressor_longformer/train/train_longformer_v2.py:81

↓ 1 callersMethod__init__

(self, items: List[Dict[str, Any]])

src/data/compressor_longformer/train/train_longformer.py:92

↓ 1 callersMethod__post_init__

(self)

llamafactory/src/llamafactory/data/formatter.py:131

↓ 1 callersFunction_apply_llama_patch

()

llamafactory/src/llamafactory/model/model_utils/longlora.py:352

↓ 1 callersFunction_bisect_in_spans

二分查找：当前位置 pos 是否落在任一 [a, b) 区间内？要求 spans 已按起点排序，且互不重叠（我们在 get_all_math_spans 里已 merge）。

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/gpt_chunk_camel_data.py:129

↓ 1 callersFunction_bisect_in_spans

二分查找：当前位置 pos 是否落在任一 [a, b) 区间内？要求 spans 已按起点排序，且互不重叠（我们在 get_all_math_spans 里已 merge）。

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/gpt_chunk_camel_data.py:129

↓ 1 callersMethod_build_messages

(self, rec: Dict[str, Any])

src/data/compressor_longformer/validate/eval_utils/evaluate.py:181

↓ 1 callersMethod_build_user_text

(self, query: str)

src/data/compressor_longformer/validate/eval_utils/evaluate.py:167

↓ 1 callersMethod_check_abnormally_long_formulas

检查异常长的公式

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/check_latex_completiness.py:109

↓ 1 callersMethod_check_abnormally_long_formulas

检查异常长的公式

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/check_latex_completiness.py:109

↓ 1 callersMethod_check_latex_pairs

检查LaTeX配对符号

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/check_latex_completiness.py:51

↓ 1 callersMethod_check_latex_pairs

检查LaTeX配对符号

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/check_latex_completiness.py:51

↓ 1 callersMethod_check_nested_formulas

检查嵌套公式问题

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/check_latex_completiness.py:136

↓ 1 callersMethod_check_nested_formulas

检查嵌套公式问题

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/check_latex_completiness.py:136

↓ 1 callersMethod_check_orphaned_symbols

检查孤立的LaTeX符号

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/check_latex_completiness.py:154

↓ 1 callersMethod_check_orphaned_symbols

检查孤立的LaTeX符号

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/check_latex_completiness.py:154

↓ 1 callersMethod_check_single_dollar_pairs

检查单$符号配对

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/check_latex_completiness.py:67

↓ 1 callersMethod_check_single_dollar_pairs

检查单$符号配对

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/check_latex_completiness.py:67

↓ 1 callersMethod_chunk_cost

(self, tok: int, boundary_feat: Sentence)

src/data/compressor/dataset_preparation/camel/camel_chunk_cot_and_answer/gpt_chunk_camel_data.py:352

↓ 1 callersMethod_chunk_cost

(self, tok: int, boundary_feat: Sentence)

src/data/compressor/dataset_preparation/camel/camel_chunk_only_cot/gpt_chunk_camel_data.py:352

↓ 1 callersFunction_collect_eos_strings

(tokenizer)

src/eval/qwen1p7b_eval/special_token_vllm.py:230

← previous next →401–500 of 1,850, ranked by callers