File tok_eval.py

scripts/tok_eval.py:None–None · view source on GitHub ↗

Source from the content-addressed store, hash-verified

1	"""
2	Evaluate compression ratio of the tokenizer.
3	"""
4

nothing calls this directly

parquets_iter_batchedFunction · 0.90

get_tokenizerFunction · 0.90

print_comparisonFunction · 0.85

from_pretrainedMethod · 0.45

get_vocab_sizeMethod · 0.45

encodeMethod · 0.45

decodeMethod · 0.45

no test coverage detected