MCPcopy Create free account
hub / github.com/apache/arrow / utf8_length_ignore_invalid

Function utf8_length_ignore_invalid

cpp/src/gandiva/precompiled/string_ops.cc:199–217  ·  view source on GitHub ↗

Count the number of utf8 characters, ignoring invalid char, considering size 1

Source from the content-addressed store, hash-verified

197
198// Count the number of utf8 characters, ignoring invalid char, considering size 1
199FORCE_INLINE
200gdv_int32 utf8_length_ignore_invalid(const char* data, gdv_int32 data_len) {
201 int char_len = 0;
202 int count = 0;
203 for (int i = 0; i < data_len; i += char_len) {
204 char_len = utf8_char_length(data[i]);
205 if (char_len == 0 || i + char_len > data_len) { // invalid byte or incomplete glyph
206 // if invalid byte or incomplete glyph, ignore it
207 char_len = 1;
208 }
209 for (int j = 1; j < char_len; ++j) {
210 if ((data[i + j] & 0xC0) != 0x80) { // bytes following head-byte of glyph
211 char_len += 1;
212 }
213 }
214 ++count;
215 }
216 return count;
217}
218
219// Get the byte position corresponding to a character position for a non-empty utf8
220// sequence

Callers 3

lpad_utf8_int32_utf8Function · 0.85
rpad_utf8_int32_utf8Function · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected