MCPcopy Create free account
hub / github.com/Notgnoshi/generative / tokenize

Function tokenize

tests/test_production.py:172–173  ·  view source on GitHub ↗
(s: str)

Source from the content-addressed store, hash-verified

170
171
172def tokenize(s: str):
173 return tuple(Token(c) for c in s)
174
175
176class RuleParsingMappings(unittest.TestCase):

Callers 3

test_simpleMethod · 0.70
test_simple_delimitedMethod · 0.70
test_probabilityMethod · 0.70

Calls 1

TokenClass · 0.90

Tested by

no test coverage detected