MCPcopy Create free account
hub / github.com/Tiiny-AI/PowerInfer / codepoints_from_utf8

Function codepoints_from_utf8

unicode.h:293–300  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

291}
292
293static std::vector<uint32_t> codepoints_from_utf8(const std::string & utf8) {
294 std::vector<uint32_t> result;
295 size_t offset = 0;
296 while (offset < utf8.size()) {
297 result.push_back(codepoint_from_utf8(utf8, offset));
298 }
299 return result;
300}
301
302static std::vector<uint16_t> codepoint_to_utf16(uint32_t cp) {
303 std::vector<uint16_t> result;

Callers 4

llm_load_vocabFunction · 0.85
bpe_gpt2_preprocessMethod · 0.85
llama_decode_textFunction · 0.85
mainFunction · 0.85

Calls 3

codepoint_from_utf8Function · 0.85
sizeMethod · 0.45
push_backMethod · 0.45

Tested by 1

mainFunction · 0.68