Print arguments.

Print arguments.

Call LLM with retry logic.

Install handlers for SIGBUS and SIGSEGV in the current process.

Compute aggregated scoring statistics.

Extract the class list from an item.

Load a JSONL file.

Rename a single key from inner to outer format.

Score a single constraint dimension. Args: item: Test data entry (contains input, output, origin_text, meta_data, etc.) cls:

Monkey-patch HYV3TopKRouter.forward to align gate input dtype with weight dtype.

Call LLM API (OpenAI-compatible endpoint).

Copy tokenizer files from tokenizer_dir to checkpoint_dir.

Parse JSON from LLM output text.

Batch scoring. Args: test_data: List of test data items responses: {(md5, instruction_lang): response_text} mapping

Check that all expected keys exist in the weight_map.

Check model.safetensors.index.json for structure and completeness.

Check the distribution of keys across shards.

Check that all referenced shard files exist and are non-empty.

Compose multi-dimension scores: final = gate_score × avg(continuous_scores)

Load a single shard, rename keys, and fuse experts. For expert groups in cross_shard_prefixes, the per-expert keys are kept as-is (just renam

Load config.json and extract model parameters.

Load model output file, returning a {(md5, instruction_lang): response} mapping. Also builds a fallback {md5: response} mapping for entries w

Entry point: called by torchrun in each worker process. Since train_lf.sh launches us via torchrun directly, all patches (template registrati

Entry point: called by torchrun in each worker process. Since train_lf_dense.sh launches us via torchrun directly, all patches (template regi

Make dataset and collator for supervised fine-tuning.

Make dataset and collator for supervised fine-tuning.

Merge cross-shard expert groups (v2: shard-centric approach). Instead of iterating per-prefix (which causes the same shard to be loaded/saved

Print scoring summary.

Pre-scan index.json to find expert groups that span multiple shards. Returns: cross_shard_prefixes: set of expert prefixes that span mult

Glossary LLM Judge (binary 0/1).

Score a single data item. Args: item: Test data entry model_response: Model output Returns: Scoring result dict

Style/Background LLM Judge (0-5 scale, normalized to 0-1).

Spot-check a few shards by loading and verifying tensor shapes.

Code/tag preservation check: verifies model_response contains all extracted_assets.

Glossary rule check: verifies that model_response contains the correct target terms from term_dict. For multi-candidate scenarios, cross-vali

Layout preservation check: verifies chunk count consistency after delimiter splitting.

Structured data validation: checks if translation preserves the original data structure.

Args: tokenizer_dir: Path to the original tokenizer directory. If None, will try to use model_name_or

Handle fatal signals (SIGBUS, SIGSEGV) by logging before exit. These signals cannot be caught by try/except. This handler ensures the error m

Wrap run_sft to inject HYV3PatchCallback.

Wrap run_sft to inject HYV3PatchCallback for tokenizer copy.

Initializer for multiprocessing pool workers. Installs signal handlers so that Bus errors in worker processes are also logged before the proc

Worker function: convert a single shard and save to output dir. Args: args_tuple: (idx, num_shards, shard_file, input_dir, output_dir, cr