MCPcopy Create free account
hub / github.com/Tiiny-AI/PowerInfer / utf8_len

Function utf8_len

llama.cpp:121–125  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

119//
120
121static size_t utf8_len(char src) {
122 const size_t lookup[] = { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 3, 4 };
123 uint8_t highbits = static_cast<uint8_t>(src) >> 4;
124 return lookup[highbits];
125}
126
127static void replace_all(std::string & s, const std::string & search, const std::string & replace) {
128 std::string result;

Callers 3

llm_load_vocabFunction · 0.70
tokenizeMethod · 0.70
tokenizeMethod · 0.70

Calls

no outgoing calls

Tested by

no test coverage detected