Scan interchange-valid UTF-8 bytes and detect most likely language If the input is in fact not valid UTF-8, this returns immediately with the result v

BIGRAM, using hash table, always advancing by 1 char Caller supplies table, such as &kCjkBiTable_obj or &kGibberishTable_obj Score all bigrams in isrc

Find top two langs and scores for one word; underpins delta tables

Take a set of <key, value> pairs and tote them up. After explicitly sorting, retrieve top key, value pairs 0xFFFF in key signifies unused

Len may be 0, for example as the low piece of length=64

Given the Language, returns its Language enum spelling, for use by programs that create C declarations, e.g. "KOREAN" "UNKNOWN_LANGUAGE" if the input

Given the Language, returns its string name used as the output by the lang/enc identifier, e.g. "Korean" "invalid_language" if the input is invalid. T

Add one more entry to copy one byte off the end, then flush

Debug output string of two octagrams, including underscores

Copy next run of same-script non-tag letters to buffer [NUL terminated] Buffer ALWAYS has leading space and trailing space space space NUL

Copy next run of non-tag characters to buffer [NUL terminated] This just replaces tags with space or \n and removes entities. Tags <br> <p> and <tr> a

Not sure who calls this... Return the probability for given language, or 0

Debug output string of one or two formatted quadgram langprobs

At this point, The chunk is described by hitbuffer->base[cspan->chunk_base .. cspan->chunk_base + cspan->base_len) hitbuffer->delta[cspan->chunk_delta

Return true if just lang1 is there: lang2=0 and lang3=0

n is in 0..3. Trailing entries are filled with UNKNOWN_LANGUAGE (which never participates in language recognition)

Force Latin, Cyrillic, Armenian, Greek scripts to be lowercase List changes with each version of Unicode, so just always lowercase Unicode 6.2.0: ARME

Maps byte offset in most recent GetOneScriptSpan/Lower span->text [0..text_bytes] into an additional byte offset from span->offset, to get back to cor

Map an offset in A to the corresponding offset in A'

Flush all if necessary

Move active window one range to the left Return true if move was OK

OCTAGRAM wrapper with surrounding underscores (offline use) Pick up 1..24 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byt

Constructor, destructor

Parse next range, 1..5 bytes Return subscript just off the end of that

Parse previous range, 1..5 bytes Return current subscript

Quick debugging dump; does not parse multi-byte items, so just length & 0x3f

Print map to file, for debugging

Take three packed languages and three probabilities 1..12 and put into uint32 For offline construction of tables

Take uint32 and unpack into three packed languages and three probabilities For runtime use of tables

QUADGRAM wrapper with surrounding underscores (offline use) Pick up 1..12 bytes plus pre/post '_' and hash them via mask/shift/add OVERSHOOTS up to 3

Get to the first real non-tag letter or entity that is a letter Sets script of that letter Return len if no more letters

We provide non-explicit singleton constructors so users can pass in a "const char*" or a "string" wherever a "StringPiece" is expected.

For testing only -- force a mapping

Take a set of <key, value> pairs and tote them up. After explicitly sorting, retrieve top key, value pairs Normal use is key=per-script language and v

Trim language tag string to canonical form for each language Input is from GetLangTagsFromHtml(), already lowercased

If the input is out of range or otherwise unrecognized, it is treated as UNKNOWN_ULSCRIPT (which never participates in language recognition)

Look up property of one UTF-8 character and advance over it Return 0 if input length is zero Return 0 and advance one byte if input is ill-formed

Scan a UTF-8 stringpiece based on state table with two-byte entries, copying to output stringpiece and doing text replacements. Also writes an optiona

Scan a UTF-8 stringpiece based on state table. Always scan complete UTF-8 characters Set number of bytes scanned. Return reason for exiting OPTIMIZED

BigOneByte versions are needed for tables > 240 states, but most won't need the TwoByte versions.

TwoByte versions are needed for tables > 240 states

Adjust a stringpiece to encompass complete UTF-8 characters. The data pointer will be increased by 0..3 bytes to get to a character boundary, and the

Inits boosts