MCPcopy Index your code
hub / github.com/zai-org/CodeGeeX / map_tokens

Method map_tokens

codegeex/mindspore/src/code_tokenizer.py:123–124  ·  view source on GitHub ↗
(self, tokens)

Source from the content-addressed store, hash-verified

121 return self.index(token)
122
123 def map_tokens(self, tokens):
124 return [self.map_token(token) for token in tokens]
125
126 def decode_tokens(self, tokens):
127 decoded = [self.string(token) for token in tokens]

Callers 1

encode_codeMethod · 0.80

Calls 1

map_tokenMethod · 0.95

Tested by

no test coverage detected