size of a valid UTF8 can be determined by looking at leading 4 bits of BYTE1 utf8_byte_size_table[0..7] --> pure ascii chars --> 1B length utf8_byte_size_table[8..11] --> internal bytes --> 1B length utf8_byte_size_table[12,13] --> 2B long UTF8 chars utf8_byte_size_table[14] --> 3B long UTF8 chars utf8_byte_size_table[15] --> 4B long UTF8 chars NOTE: Results for invalid/ malformed utf-8 sequences
| 289 | // NOTE: Results for invalid/ malformed utf-8 sequences are undefined. |
| 290 | // ex: \xFF... returns 4B |
| 291 | static inline uint8_t ValidUtf8CodepointByteSize(const uint8_t* codeunit) { |
| 292 | return internal::utf8_byte_size_table[*codeunit >> 4]; |
| 293 | } |
| 294 | |
| 295 | static inline bool Utf8IsContinuation(const uint8_t codeunit) { |
| 296 | return (codeunit & 0xC0) == 0x80; // upper two bits should be 10 |