For now, long long is 64-bit on all the platforms we care about, so these functions can simply pass the call to strto[u]ll.

Back up one UTF-8 character

BIGRAM Pick up 1..8 bytes and hash them via mask/shift/add. NO pre/post OVERSHOOTS up to 3 bytes For runtime use of tables

Remove portions of text that have a high density of spaces, or that are overly repetitive, squeezing the remaining text in-place to the front of the i

Demote all languages except Top40 and plus_one Do this just before sorting chunk_tote results

Detects a language of the UTF-16 encoded zero-terminated text. Returns: Language enum.

Result vector must be exactly three items

Return version text string String is "code_version - data_scrape_date"

BIGRAM, using hash table, always advancing by 1 char Caller supplies table, such as &kCjkBiTable_obj or &kGibberishTable_obj Score all bigrams in isrc

OCTAGRAM, using hash table, always advancing by 1 word Caller supplies table, such as &kLongWord8Table_obj Score all words in isrc, using languages th

QUADGRAM, using hash table, advancing by 2/4/8/16 chars Caller supplies table, such as &kQuadTable_obj or &kGibberishTable_obj Score up to n quadgrams

UNIGRAM, using UTF-8 property table, advancing by 1/2/4/8 chars Caller supplies table, such as compact_lang_det_generated_ctjkvz_b1_obj Score up to n

Src points to '&' Writes entity value to dst. Returns take(src), put(dst) byte counts

Given the Language, returns its Language enum spelling, for use by programs that create C declarations, e.g. "KOREAN" "UNKNOWN_LANGUAGE" if the input

Extract representative pattern of exactly N values into dst[0..n-1] This routine may be called multiple times, but it may downsample as a side effect,

Collapse tail end by simple median across disparate-weight values, dropping or duplicating last value if need be. This routine is idempotent.

Convert "en-Latn-GB" to ULScript_Latin

Convert "en-Latn-GB" to ENGLISH Normalize to PORTUGUESE, not PORTUGUESE_B nor PORTUGUESE_P Consider for later: NORWEGIAN, NORWEGIAN_N Consider for lat

Calculate ratio of score per 1KB vs. expected score per 1KB

Copy next run of same-script non-tag letters to buffer [NUL terminated] Buffer has leading space and all text is lowercased

Calculate reliablity of len bytes of script lscript with chunk_tote

Gets lscript number for letters; always returns 0 (common script) for non-letters

Useful single-byte tests

Return one of five probabilities in an entry CURRENTLY UNUSED

A non-macro interface to the log facility; (useful when the logging level is not a compile-time constant).

Force Latin, Cyrillic, Greek scripts to be lowercase

Return median of seq_[sub] .. seq_[sub+2], favoring middle element

Returns "0" for too small

Compress level k by 3x, creating level k+1

Creates fully initialized NormalizationAPI object. Loads DLL and binds all referenced functions.

Merge together some languages, such as bo/hr/sr Croatian Latin and Serbian Cyrillic now.

Initialize the LogStream so the messages can be written somewhere (they'll never be actually displayed). This will be needed if a NullStream& is impli

OCTAGRAM wrapper with surrounding spaces Pick up 1..24 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byte, OVERSHOOTS up to

OCTAGRAM wrapper with surrounding underscores (offline use) Pick up 1..24 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byt

BIGRAM, QUADGRAM, OCTAGRAM score one => tote Input: 4-byte entry of 3 language numbers and one probability subscript, plus an accumulator tote. (langu

UNIGRAM score one => tote Input: 1-byte entry of subscript into unigram probs, plus an accumulator tote. Output: running sums in tote updated

QUADGRAM wrapper with surrounding spaces Pick up 1..12 bytes plus pre/post space and hash them via mask/shift/add UNDERSHOOTS 1 byte, OVERSHOOTS up to

QUADGRAM wrapper with surrounding underscores (offline use) Pick up 1..12 bytes plus pre/post '_' and hash them via mask/shift/add OVERSHOOTS up to 3

Return reliablity of result 0..100 for top two scores delta==0 is 0% reliable, delta==fully_reliable_thresh is 100% reliable (on a scale where +1 is a

Return reliablity of result 0..100 for top score vs. mainsteam score Values are score per 1024 bytes of input ratio = max(top/mainstream, mainstream/t

Construct a logging site from a logging level and epoch.

Get to the first real non-tag letter or entity that is a letter Sets script of that letter Return len if no more letters

Detect language of plaintext src

Detect extended language of plaintext src

Take a set of <key, value> pairs and tote them up. After explicitly sorting, retrieve top key, value pairs

Take a set of <key, value> pairs and tote them up. After explicitly sorting, retrieve top key, value pairs

Determine whether verbose logging should occur at a given log site. TODO(llansing): Find a way to eliminate FLAGS_silent_init from this function whil

Although it returns an int, atoi() is implemented in terms of strtol, and so has differing overflow and underflow behavior. atol is the same.