MCPcopy Create free account
hub / github.com/THUDM/GLM / prep_command_tokens

Function prep_command_tokens

data_utils/tokenization.py:150–151  ·  view source on GitHub ↗
(tokenlist, token_format=token_format)

Source from the content-addressed store, hash-verified

148
149
150def prep_command_tokens(tokenlist, token_format=token_format):
151 return [CommandToken(tok[0], token_format.format(tok[0]), tok[1]) for tok in tokenlist]
152
153
154class CommandToken(object):

Callers 1

tokenization.pyFile · 0.85

Calls 1

CommandTokenClass · 0.85

Tested by

no test coverage detected