MCPcopy Create free account
hub / github.com/THUDM/GLM / insert

Method insert

data_utils/tokenization.py:99–111  ·  view source on GitHub ↗
(self, idx, other)

Source from the content-addressed store, hash-verified

97 return len(self.tokenization)
98
99 def insert(self, idx, other):
100 if isinstance(other, (CommandToken, TypeToken)):
101 self.tokenization.insert(idx, other.Id)
102 if idx == 0:
103 self.text = other.token + self.text
104 self.original_text = other.token + self.original_text
105 elif idx == len(self.tokenization) - 1:
106 self.text += other.token
107 self.original_text += other.token
108 elif isinstance(other, Tokenization):
109 self.tokenization = self.tokenization[:idx] + other.tokenization + self.tokenization[idx:]
110 else:
111 self.tokenization = self.tokenization[:idx] + other.tokenization + self.tokenization[idx:]
112
113 def append(self, other):
114 if isinstance(other, (CommandToken, TypeToken)):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected