MCPcopy Create free account
hub / github.com/THUDM/GLM / convert_token_to_id

Method convert_token_to_id

data_utils/sp_tokenizer.py:123–124  ·  view source on GitHub ↗
(self, token)

Source from the content-addressed store, hash-verified

121 return [self.sp.PieceToId(token) for token in tokens]
122
123 def convert_token_to_id(self, token):
124 return self.sp.PieceToId(token)
125
126 def convert_id_to_token(self, idx):
127 return self.sp.IdToPiece(idx)

Callers 1

TokenToIdMethod · 0.80

Calls

no outgoing calls

Tested by

no test coverage detected