File tokenizer.py

python/tokenizer.py:None–None · view source on GitHub ↗

Source from the content-addressed store, hash-verified

1	import gzip
2	import html
3	import os
4	from functools import lru_cache

nothing calls this directly

SimpleTokenizerClass · 0.70

decodeMethod · 0.45

encodeMethod · 0.45

no test coverage detected