Return language and script from parsed line

Return position of start of text

Return position of start of text

Debug output string of one unigram

Score up to 64KB of a single script span in one pass Make a dummy entry off the end to calc length of last span Return offset of first unused input by

Add just one element to resultchunk vector: For RTypeNone or RTypeOne

Effectively, do a merge-sort based on text offsets Look up each indirect value in appropriate scoring table and keep just the resulting langprobs

Merge-sort the individual hit arrays, go indirect on the scoring subscripts, break linear array into chunks. Input: hitbuffer base, delta, distinct a

Useful for converting an entity to an ascii value. RETURNS unicode value, or -1 if entity isn't valid. Don't include & or ;

Move all the text bytes from lower byte-count to higher one

From 40-bit gram FP, return hash table subscript and remaining key

OCTAGRAM wrapper with surrounding spaces Pick up 1..24 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byte, OVERSHOOTS up to

Hash a consecutive pair of tokens/words A B Old: hash is B - A, which gives too many false hits on one-char diffs Now: rotate(A,13) + B

From 32-bit gram FP, return hash table subscript and remaining key

src points to a letter. Find the byte length of a quadgram starting there.

Unescape the current character pointed to by src. SETS the number of chars read for the conversion (in UTF8). If src isn't a valid entity, just cons

Move less likely byte count to more likely for close pairs of languages If given, also update resultchunkvector

Return reliablity of result 0..100 for top two scores delta==0 is 0% reliable, delta==fully_reliable_thresh is 100% reliable (on a scale where +1 is a

Delete any extended languages from doc_tote

Move bytes for unreliable langs to another lang or UNKNOWN doc_tote is sorted, so cannot Add If both CHINESE and CHINESET are present and unreliable,

Score chunks of text described by hitbuffer, allowing each to be in a different language, and optionally adjusting the boundaries inbetween. Set last_

For each chunk, add extra weight for language priors (from content-lang and meta lang=xx) and distinctive tokens

Score RTypeCJK scriptspan into doc_tote and vec, updating scoringcontext

Score RTypeNone or RTypeOne scriptspan into doc_tote and vec, updating scoringcontext

Score all the bases, deltas, distincts, boosts for one chunk into chunk_tote After last chunk there is always a hitbuffer entry with an offset just of

Return score per 1024 bytes for top language

Score one scriptspan into doc_tote and vec, updating scoringcontext Inputs: One scriptspan of perhaps 40-60KB, all same script lower-case letters and

Score RTypeMany scriptspan into doc_tote and vec, updating scoringcontext We have a scriptspan with all lowercase text in one script. Look up quadgram

Add hints to vector of langpriors Input is string after HTTP header Content-Language:

Add hints to vector of langpriors Input is from DetectEncoding()

Add hints to vector of langpriors Input is from random source

Add hints to vector of langpriors Input is last element of hostname (no dot), e.g. from GetTLD()

Needs expected score per 1KB in scoring context

For all but the first summary, if its top language differs from the previous chunk, refine the boundary Linearized version

Add n elements of summarybuffer to resultchunk vector: Each element is letters-only text [offset..offset+bytes) This maps back to original[Back(offset

Remove any trailing digits/spaces (possible mapreduce counts) Return length

Trim language priors to no more than max_entries, keeping largest abs weights

BigOneByte versions are needed for tables > 240 states, but most won't need the TwoByte versions. Internally, to next-to-last offset is multiplied by

TwoByte versions are needed for tables > 240 states Look up property of one UTF-8 character and advance over it Return 0 if input length is zero Retur

Scan a UTF-8 stringpiece based on state table, copying to output stringpiece and doing text replacements. Also writes an optional OffsetMap. Pass NULL

Scan a UTF-8 stringpiece based on state table, copying to output stringpiece and doing text replacements. DO NOT CALL DIRECTLY. Use UTF8GenericReplace

Scan a UTF-8 stringpiece based on state table with two-byte entries, copying to output stringpiece and doing text replacements. DO NOT CALL DIRECTLY.

src points to a letter. Find the byte length of a unigram starting there.

Assuming that all fields have been set in the specified header, re-align the starting positions of all data chunks to be aligned along 64-bit boundari

As noted on http://stackoverflow.com/questions/1001307, gcc is highly likely to convert this function's return into a constant - meaning that any if-b

runetochar copies (encodes) one rune, pointed to by r, to at most UTFmax bytes starting at s and returns the number of bytes generated.

Re-written (simplified) from BSD strings.h

Update scoring context distinct_boost for single language quad

Converts a unigram prob/lang byte into an approximate prob/lang triple Just keeps the largest value. Now unused.

Back up over previous range, 1..5 bytes Return subscript at the beginning of that. Pins at 0

For showing one chunk Print debug output for one scored chunk Optionally print out per-chunk scoring information In degenerate cases, hitbuffer and cs

For showing all chunks

For showing all chunks