Add to mapping from A to A', specifying how many next bytes are identical in A and A'

Return one of three probabilities in an entry

Returns mid if key found in lo <= mid < hi, else -1

Add to mapping from A to A', specifying how many next bytes are deleted from A while not advancing in A' at all

data() may return a pointer to a buffer with embedded NULs, and the returned buffer may or may not be null terminated. Therefore it is typically a mi

Skips over xxxxx_ where _ is one or more spaces/tabs Returns string::npos if no more fields

Skips over xxxxx_ where _ is one or more spaces/tabs Returns string::npos if no more fields

Results language3/percent3/text_bytes must be exactly three items

Return score for a particular per-script language, or zero

Scan forwards for case-insensitive string s in [pos..max_pos) Bytes of s must already be lowercase, i.e. in [20..3f] or [60..7f] Cheap lowercase. Cont

Returns which set of statistically-close languages lang is in. 0 means none.

sum lg prob

Reading exposed here; setting hidden in .cc

Add to mapping from A to A', specifying how many next bytes are inserted in A' while not advancing in A at all

Clear the map After: next_diff_sub_ is 0 Windows are the a and a' ranges covered by diffs_[next_diff_sub_-1] which is a fake range of width 0 mapping

Debug output string of one formatted langprob Returns "en.24&nbsp;fr.10&nbsp;es.4"

Gets lscript number for letters; always returns 0 (common script) for non-letters

Return address of 8-byte entry[i]

Merge in another language prior, boosting 10x if already there

Return true if current Tbl pointer is within state0 range Note that unsigned compare checks both ends of range simultaneously

Look up 32-bit gram FP in caller-passed table Typical size 256K entries (1.5MB) Two-byte hashkey

These are for ASCII-only tag names Compare one letter uplow to c, ignoring case of uplowp

Scan backwards for case-insensitive string s in [min_pos..pos) Bytes of s must already be lowercase, i.e. in [20..3f] or [60..7f] Cheap lowercase. Con

Reset to offset 0

byte count

These are for ASCII-only tag names Return true for space \n false for \r

Binary lookup on language tag

Src points to '&' Writes entity value to dst. Returns take(src), put(dst) byte counts

Returns true if character is < > or &, none of which are letters

Map an offset in A' to the corresponding offset in A

Merge in another language prior, taking max if already there

Counts number of spaces; a little faster than one-at-a-time Doesn't count odd bytes at end

Avoid zdiv

Debug output string of one formatted pslang,qprob pair

Name can be either full name or ISO code, or can be ISO code embedded in a language-script combination such as "ABKHAZIAN", "en", "en-Latn-GB"

Return internal probability score (sum) per 1024 bytes

Return true if language can be in the Latin script

Return true if language can be in a non-Latin script

Map script into Latin, Cyrillic, Arabic, Other

Create a langprob packed value from its parts. qprob is quantized [0..12] We use Latn script to represent any RTypeMany language

These are for ASCII-only tag names Return true for space / < > etc. all less than 0x40

Look up 40-bit gram FP in caller-passed table Typical size 256K-4M entries (1-16MB) 24-12 bit hashkey packed with 8-20 bit indirect lang/probs keymask

src points to a letter. Find the byte length of an octagram starting there.

src points to non-letter, such as tag-opening '<' Return length from here to next possible letter On another < before >, return 1 advances <tag> |

Scan a UTF-8 stringpiece based on state table. Always scan complete UTF-8 characters Set number of bytes scanned. Return reason for exiting

Backscan to word boundary, returning how many bytes n to go back so that src - n is non-space ans src - n - 1 is space. If not found in kMaxSpaceScan

BIGRAM Pick up 1..8 bytes and hash them via mask/shift/add. NO pre/post OVERSHOOTS up to 3 bytes For runtime use of tables Does X86 unaligned loads

For showing one chunk Print debug output for one scored chunk Optionally print out per-chunk scoring information In degenerate cases, hitbuffer and cs