MCPcopy Create free account
hub / github.com/FSoft-AI4Code/CodeText-parser / tokenize_code

Function tokenize_code

src/codetext/parser/language_parser.py:24–30  ·  view source on GitHub ↗
(node, blob: str, nodes_to_exclude: Optional[Set]=None)

Source from the content-addressed store, hash-verified

22
23
24def tokenize_code(node, blob: str, nodes_to_exclude: Optional[Set]=None) -> List:
25 tokens = []
26 traverse(node, tokens)
27 # print(tokens)
28 # for token in tokens:
29 # print(token.text)
30 return [match_from_span(token, blob) for token in tokens if nodes_to_exclude is None or token not in nodes_to_exclude]
31
32def nodes_are_equal(n1, n2):
33 return n1.type == n2.type and n1.start_point == n2.start_point and n1.end_point == n2.end_point

Callers

nothing calls this directly

Calls 2

traverseFunction · 0.85
match_from_spanFunction · 0.85

Tested by

no test coverage detected