MCPcopy Create free account
hub / github.com/THUDM/GLM / parse_command_tokens

Method parse_command_tokens

data_utils/tokenization.py:84–91  ·  view source on GitHub ↗
(self)

Source from the content-addressed store, hash-verified

82 return self.parse_command_tokens()
83
84 def parse_command_tokens(self):
85 if self.command_tokens is None:
86 return
87 for command_token in self.command_tokens:
88 if self.asIds:
89 setattr(self, command_token.name, command_token.Id)
90 else:
91 setattr(self, command_token.name, command_token.token)
92
93 def __getitem__(self, index):
94 return self.tokenization[index]

Callers 2

__init__Method · 0.95
set_command_tokensMethod · 0.95

Calls

no outgoing calls

Tested by

no test coverage detected