MCPcopy Create free account
hub / github.com/apache/arrow / ValidUtf8CodepointByteSize

Function ValidUtf8CodepointByteSize

cpp/src/arrow/util/utf8_internal.h:291–293  ·  view source on GitHub ↗

size of a valid UTF8 can be determined by looking at leading 4 bits of BYTE1 utf8_byte_size_table[0..7] --> pure ascii chars --> 1B length utf8_byte_size_table[8..11] --> internal bytes --> 1B length utf8_byte_size_table[12,13] --> 2B long UTF8 chars utf8_byte_size_table[14] --> 3B long UTF8 chars utf8_byte_size_table[15] --> 4B long UTF8 chars NOTE: Results for invalid/ malformed utf-8 sequences

Source from the content-addressed store, hash-verified

289// NOTE: Results for invalid/ malformed utf-8 sequences are undefined.
290// ex: \xFF... returns 4B
291static inline uint8_t ValidUtf8CodepointByteSize(const uint8_t* codeunit) {
292 return internal::utf8_byte_size_table[*codeunit >> 4];
293}
294
295static inline bool Utf8IsContinuation(const uint8_t codeunit) {
296 return (codeunit & 0xC0) == 0x80; // upper two bits should be 10

Callers 1

TransformMethod · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected