File verify_tokenizer.py

scripts/verify_tokenizer.py:None–None · view source on GitHub ↗

Source from the content-addressed store, hash-verified

1	#!/usr/bin/env python3
2	"""Generate ground-truth BPE tokenizations for test_tokenizer.cpp verification.
3
4	Usage:

nothing calls this directly

mainFunction · 0.70

no test coverage detected