MCPcopy Create free account
hub / github.com/InternLM/InternLM / alpaca_tokenizer.py

File alpaca_tokenizer.py

tools/alpaca_tokenizer.py:None–None  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1import argparse
2import json
3import os.path as osp
4from pathlib import Path

Callers

nothing calls this directly

Calls 2

dump_bin_meta_binFunction · 0.85
processFunction · 0.70

Tested by

no test coverage detected