MCPcopy Create free account
hub / github.com/YunwenTechnology/QueryGeneration / tokenize

Method tokenize

tokenization/tokenization.py:183–205  ·  view source on GitHub ↗

Tokenizes a piece of text.

(self, text)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers

nothing calls this directly

Calls 6

_clean_textMethod · 0.95
_tokenize_chinese_charsMethod · 0.95
_run_strip_accentsMethod · 0.95
_run_split_on_puncMethod · 0.95
convert_to_unicodeFunction · 0.85
whitespace_tokenizeFunction · 0.85

Tested by

no test coverage detected