Given the Language, return the language code, e.g. "ko"

Three-way associative, guaranteeing that the largest two counts are always in the data structure. kMaxSize must be a multiple of 3, and is tied to the

Make languages packed into uint32 values non-zero These routines later could remap so languages not in QuadHash tables are not represented, and so tha

Compare the two strings s1 and s2 without regard to case using the current locale; returns 0 if they are equal, 1 if s1 > s2, and -1 if s2 > s1 accord

Return one of three probabilities in an entry

Given the Language, returns its string name used as the output by the lang/enc identifier, e.g. "Korean" "invalid_language" if the input is invalid. T

Returns pointer to the normalized text.

------------------------------------------------------------------------------ General ---------------------------------------------------------------

IsValidLanguage =============== Function to check if the input is within range of the Language enum. If IsValidLanguage(lang) returns true, it is saf

Return length of a single UTF-8 source character

Reset. Deletes the current owned object, if any. Then takes ownership of a new object, if given. this->reset(this->get()) works.

Returns true if all functions were bound successfully. This implies that library_ itself was loaded successfully.

Return address of 8-byte entry[i]

Add one chunk's score to running document score If the top language is UNKNOWN_LANGUAGE, score nothing. This is used to positively identify text to be

Increment count of quadgrams/trigrams/unigrams scored

Src points to '&' Writes entity value to dst. Returns take(src), put(dst) byte counts

Returns true if character is < > or &

Sort first n entries by decreasing order of value If key==0 other fields are not valid, treat value as -1

Swap two scoped pointers.

This uses a cheap predictor to get a measure of compression, and hence a measure of repetitiveness. It works on complete UTF-8 characters instead of b

Counts number of spaces; a little faster than one-at-a-time Doesn't count odd bytes at end

Return current top key

Extract return values before fixups

Return true if current Tbl pointer is within state0 range Note that unsigned compare checks both ends of range simultaneously

Start the tote with a count of one for the default language for script

Map script into Latin, Cyrillic, Arabic, Other. Used in keeping track of amount of training data for language-script combinations

OCTAGRAM Pick up 1..24 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byte, OVERSHOOTS up to 3 bytes The low 32 bits follow

QUADGRAM Pick up 1..12 bytes plus pre/post space and hash them via mask/shift/add OVERSHOOTS up to 3 bytes For runtime use of tables

Look up 32-bit gram FP in caller-passed table Typical size 256K entries (1.5MB) Two-byte hashkey

Quick Skip to next letter or < > & or to end of string (eos) Always return is_letter for eos

src points to non-letter, such as tag-opening '<' Return length from here to next possible letter On eos or another < before >, return 1 advances <tag

Add one chunk's score to running document score Convenience function with constant src text

This implementation is not as good as the one in base/strtoint.h, but it's sufficient for our purposes.

Increment the initial probabilities based on a per-encoding probs entry

Increment the initial probability for given language by fixed amount Does not recognize extended languages as hints

Increment the initial probabilities based on a per-TLD probs entry

Backscan to word boundary, returning how many bytes n to go back so that src - n is non-space ans src - n - 1 is space. If not found in kMaxSpaceScan

Calculate a single summary language for the document, and its reliability. Returns language3[0] or language3[1] or ENGLISH or UNKNOWN_LANGUAGE This is

Remove words of text that have more than half their letters predicted correctly by our cheap predictor, moving the remaining words in-place to the fro

Timing 2.8GHz P4 (dsites 2008.03.20) with 170KB input About 90 MB/sec, with or without memcpy, chunksize 48 or 4096 Just CountSpaces is about 340 MB/s

We do not use it, just to please a compiler and minimize ported code changes.

Find subscript of a given packed language, or -1

Forwardscan to word boundary, returning how many bytes n to go forward so that src + n is non-space ans src + n - 1 is space. If not found in kMaxSpac

Copy next run of same-script non-tag letters to buffer [NUL terminated] Force Latin and Cyrillic scripts to be lowercase

The global epoch is the least significant half of an Atomic32, and may only be accessed through atomic operations.

Find subscript of matching key in first 4 bytes of sorted hint array, or -1

Return true if the language is "unknown". (This function was previously a macro, hence the spelling in all caps.)

Proxy functions for the ones loaded from DLL.

Normalize ASCII string to first 4 alphabetic/digit chars Letters are forced to lowercase ASCII Used to normalize TLD values

From 40-bit gram FP, return hash table subscript and remaining key

Look up 40-bit gram FP in caller-passed table Typical size 256K-4M entries (1-16MB) 24-12 bit hashkey packed with 8-20 bit indirect lang/probs keymask

From 32-bit gram FP, return hash table subscript and remaining key

Move less likely byte count to more likely for close pairs of languages

Delete any extended languages from doc_tote

Move bytes for unreliable langs to another lang or UNKNOWN doc_tote is sorted, so cannot Add If both CHINESE and CHINESET are present and unreliable,

Score one scriptspan using the only language for that script

Score one scriptspan using quadgrams Incoming chunk_tote may have partial accumulation

Score one scriptspan using unigrams Updates tote_grams

The least significant half of a site is the epoch.

The most significant half of a site is the logging level.

Look up property of one UTF-8 character and advance over it Return 0 if input length is zero Return 0 and advance one byte if input is ill-formed

BigOneByte versions are needed for tables > 240 states, but most won't need the TwoByte versions. Internally, to next-to-last offset is multiplied by

Scan a UTF-8 stringpiece based on a state table. Always scan complete UTF-8 characters Set number of bytes scanned. Return reason for exiting

Conversions to a 32-bit integer can pass the call to strto[u]l on 32-bit platforms, but need a little extra work on 64-bit platforms.