MCPcopy Create free account
hub / github.com/ACL2020SpellGCN/SpellGCN / _tokenize_chinese_chars

Method _tokenize_chinese_chars

tokenization.py:252–263  ·  view source on GitHub ↗

Adds whitespace around any CJK character.

(self, text)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 1

tokenizeMethod · 0.95

Calls 1

_is_chinese_charMethod · 0.95

Tested by

no test coverage detected