MCPcopy Create free account
hub / github.com/0xShug0/audio.cpp / unicode_cpt_flags_from_utf8

Function unicode_cpt_flags_from_utf8

external/llama_tokenizer/unicode.cpp:1020–1027  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1018}
1019
1020unicode_cpt_flags unicode_cpt_flags_from_utf8(const std::string & utf8) {
1021 static const unicode_cpt_flags undef(unicode_cpt_flags::UNDEFINED);
1022 if (utf8.empty()) {
1023 return undef; // undefined
1024 }
1025 size_t offset = 0;
1026 return unicode_cpt_flags_from_cpt(unicode_cpt_from_utf8(utf8, offset));
1027}
1028
1029std::string unicode_byte_to_utf8(uint8_t byte) {
1030 static std::unordered_map<uint8_t, std::string> map = unicode_byte_to_utf8_map();

Callers

nothing calls this directly

Calls 3

unicode_cpt_from_utf8Function · 0.85
emptyMethod · 0.45

Tested by

no test coverage detected