MCPcopy Create free account
hub / github.com/THUDM/GLM / IdToToken

Method IdToToken

data_utils/tokenization.py:1224–1234  ·  view source on GitHub ↗
(self, Id, type_token=False)

Source from the content-addressed store, hash-verified

1222 # return Tokenization(tokens, processed_text, text, asIds=False)
1223
1224 def IdToToken(self, Id, type_token=False):
1225 if isinstance(Id, (TypeToken, CommandToken)):
1226 return Id.token
1227 if type_token:
1228 return self.type_id_map[Id].token
1229 if Id in self.command_id_map:
1230 return self.command_id_map[Id].token
1231 elif Id in self.type_id_map:
1232 return self.type_id_map[Id].token
1233 else:
1234 return self.text_tokenizer.convert_id_to_token(int(Id))
1235
1236 def TokenToId(self, token, type_token=False):
1237 if isinstance(token, (TypeToken, CommandToken)):

Callers

nothing calls this directly

Calls 1

convert_id_to_tokenMethod · 0.80

Tested by

no test coverage detected