Copy attribute value in [pos..max_pos) pos is just after an opening quote/apostrophe and max_pos is the ending one String must all be on a single line

Return current top three keys

Scan interchange-valid UTF-8 bytes and detect list of top 3 extended languages. Extended languages are additional Google interface languages and Unico

Given the Language, return the language code, e.g. "ko"

Extract return values before fixups

Guarantees that the resulting output value is interchange valid 00-FF; map to spaces or MS CP1252 D800-DFFF; surrogates FDD0-FDEF; non-characters xxFF

Forwardscan to word boundary, returning how many bytes n to go forward so that src + n is non-space ans src + n - 1 is space. If not found in kMaxSpac

Advances src, decrements len

If next chunk language matches last vector language, extend last element Otherwise add new element to vector

Convert GetTimeOfDay output to 64-bit usec

Move active window one range to the right Return true if move was OK

OCTAGRAM Pick up 1..24 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byte, OVERSHOOTS up to 3 bytes The low 32 bits follow

If high bit is on, take out high bit and add 2B to make table2 entries easy

BIGRAM, QUADGRAM, OCTAGRAM score one => tote Input: 4-byte entry of 3 language numbers and one probability subscript, plus an accumulator tote. (langu

BIGRAM, QUADGRAM, OCTAGRAM score one => tote Input: 4-byte entry of 3 language numbers and one probability subscript, plus an accumulator tote. (langu

QUADGRAM wrapper with surrounding spaces Pick up 1..12 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byte, OVERSHOOTS up to

QUADGRAM Pick up 1..12 bytes plus pre/post space and hash them via mask/shift/add OVERSHOOTS up to 3 bytes For runtime use of tables Does X86 unaligne

Quick Skip to next letter or < > & or to end of string (eos) Always return is_letter for eos

Add hints to vector of langpriors Input is from GetLangTagsFromHtml(), already lowercased

Sort first n entries by decreasing order of value If key==0 other fields are not valid, treat value as -1

Returns the length in bytes of the prefix of src that is all interchange valid UTF-8

Update scoring context distinct_boost for distinct octagram Keep last 4 used. Since these are mostly (except at splices) in hitbuffer, we might be abl

Increment count of quadgrams/trigrams/unigrams scored

Within hitbufer->linear[] <-- prior chunk --><-- this chunk --> | | | linear0 linear1 linear2 la

src points to a letter. Find the byte length of a bigram starting there.

Calculate a single summary language for the document, and its reliability. Returns language3[0] or language3[1] or ENGLISH or UNKNOWN_LANGUAGE This is

Remove words of text that have more than half their letters predicted correctly by our cheap predictor, moving the remaining words in-place to the fro

This alternate form overwrites redundant words, thus avoiding corrupting the backmap for generating a vector of original-text ranges.

Remove portions of text that have a high density of spaces, or that are overly repetitive, squeezing the remaining text in-place to the front of the i

This alternate form overwrites redundant words, thus avoiding corrupting the backmap for generating a vector of original-text ranges.

Timing 2.8GHz P4 (dsites 2008.03.20) with 170KB input About 90 MB/sec, with or without memcpy, chunksize 48 or 4096 Just CountSpaces is about 340 MB/s

Break linear array into chunks of ~20 quadgram hits or ~50 CJK unigram hits

Find and copy attribute value: quoted string in [pos..max_pos) Return slightly-normalized language list, empty or ending in comma

Make a langprob that gives small weight to the default language for ulscript

Scan interchange-valid UTF-8 bytes and detect most likely language

Scan interchange-valid UTF-8 bytes and detect list of top 3 languages.

Return number of src bytes skipped

Binary lookup on tld

Make printable string of priors

Move this verbose debugging output to debug.cc eventually

Use this one. Hints are collected into a struct. Flags are passed in (normally zero). Also returns 3 internal language scores as a ratio to normal s

Find subscript of a given packed language, or -1

For constructing tables Given a vector of 3 probabilities 1..12, find subscript of best table match. Minimizes RMS error Brute-force version

Find closing quot/apos. Also stop on = > < and & (simplistic parsing)

Find opening quote or apostrophe, skipping spaces Note: this is all somewhat insensitive to mismatched quotes

Find closing '>' for HTML tag. Also stop on < and & (simplistic parsing)

Find opening '<' for HTML tag Note: this is all somewhat insensitive to mismatched quotes

Extend results to fully cover the [lo..hi) range

Debug output string of one bigram

Score up to 64KB of a single script span, doing both delta-bi and distinct bis in one pass

Return language and script from parsed line or defaults

Debug output string of one octagram, including underscores

Score up to 64KB of a single script span, doing both delta-octa and distinct words in one pass

Copy next run of same-script non-tag letters to buffer [NUL terminated] Force Latin, Cyrillic, Greek scripts to be lowercase Buffer ALWAYS has leading

Debug output string of one quadgram, including underscores

Score up to 64KB of a single script span in one pass Make a dummy entry off the end to calc length of last span Return offset of first unused input by