MCPcopy Create free account
hub / github.com/zai-org/CodeGeeX / push_acc_space

Function push_acc_space

codegeex/mindspore/src/code_tokenizer.py:15–23  ·  view source on GitHub ↗
(acc_len: int, text: str)

Source from the content-addressed store, hash-verified

13 """
14
15 def push_acc_space(acc_len: int, text: str):
16 if acc_len == 0:
17 return text
18 if acc_len == 1:
19 return text + ' '
20 assert acc_len <= max_len, f'Max whitespace run length {max_len}, but found {acc_len}'
21 extra_id = start_extra_id - 2 + acc_len
22 extra_token = f'<|extratoken_{extra_id}|>'
23 return text + extra_token
24
25 acc_len = 0
26 res = ''

Callers 1

encode_whitespacesFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected