Method tokenize

tokenization.py:170–176 · view source on GitHub ↗

(self, text)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

test_full_tokenizerMethod · 0.95

convert_single_exampleFunction · 0.45

no outgoing calls

test_full_tokenizerMethod · 0.76

convert_single_exampleFunction · 0.36