MCPcopy Create free account
hub / github.com/PABannier/sam3.cpp / bytes_to_unicode

Function bytes_to_unicode

scripts/verify_tokenizer.py:21–30  ·  view source on GitHub ↗
()

Source from the content-addressed store, hash-verified

19
20
21def bytes_to_unicode():
22 bs = list(range(33, 127)) + list(range(161, 173)) + list(range(174, 256))
23 cs = list(bs)
24 n = 0
25 for b in range(256):
26 if b not in bs:
27 bs.append(b)
28 cs.append(256 + n)
29 n += 1
30 return dict(zip(bs, [chr(c) for c in cs]))
31
32
33def get_pairs(word):

Callers 1

__init__Method · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected