Method tokenize

tokenization.py:196–218 · view source on GitHub ↗

Tokenizes a piece of text.

(self, text)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

test_chineseMethod · 0.95

test_basic_tokenizer_lowerMethod · 0.95

test_basic_tokenizer_no_lowerMethod · 0.95

_clean_textMethod · 0.95

_tokenize_chinese_charsMethod · 0.95

_run_strip_accentsMethod · 0.95

_run_split_on_puncMethod · 0.95

convert_to_unicodeFunction · 0.85

whitespace_tokenizeFunction · 0.85

test_chineseMethod · 0.76

test_basic_tokenizer_lowerMethod · 0.76

test_basic_tokenizer_no_lowerMethod · 0.76