MCPcopy Create free account
hub / github.com/Notgnoshi/generative / tokenize

Function tokenize

tests/test_grammar.py:12–13  ·  view source on GitHub ↗
(s: str)

Source from the content-addressed store, hash-verified

10
11
12def tokenize(s: str):
13 return [Token(c) for c in s]
14
15
16class ContextFreeParsing(unittest.TestCase):

Callers 15

test_apply_rules_aMethod · 0.70
test_apply_rules_bMethod · 0.70
test_rewrite_aMethod · 0.70
test_rewrite_bMethod · 0.70
test_rewrite_abaaMethod · 0.70
test_rewrite_abMethod · 0.70
test_loopMethod · 0.70
setUpMethod · 0.70
test_pick_rule_aMethod · 0.70
test_pick_rule_bMethod · 0.70
test_pick_rule_cMethod · 0.70
test_left_contextMethod · 0.70

Calls 1

TokenClass · 0.90

Tested by

no test coverage detected