Write writes the index x to w.

reasonable for BenchmarkSaveRestore Read reads the index from r into x; x must not be nil.

bucketMax_8_32 stores into bucket[c] the maximum index in the bucket for character c in a bucket-sort of text. The bucket indexes for c are [min, max)

Will return the prediction of the next token distribution corresponding to the longest suffix in queryIds. For a suffix to be considered valid, there

Puts the value into the byte slice at the given index.

Lookup returns an unsorted list of at most n indices where the byte string s occurs in the indexed data. If n < 0, all occurrences are returned. The r

Search for the occurrences of a query in the suffix array. Returns the starting and ending positions of the occurrences.

Perform left or right binary search on the suffix array.

bucketMin_8_32 stores into bucket[c] the minimum index in the bucket for character c in a bucket-sort of text.

Compare whether a is before b lexographically. Returns -1 if a < b, 0 if a == b, and 1 if a > b.

freq_8_32 returns the character frequencies for text, as a slice indexed by character value. If freq is nil, freq_8_32 uses and returns bucket. If fre

Retrieve the suffix array for a given index.

Create a multi-suffix array from a list of suffix array paths.

map_32 maps the LMS-substrings in text to their new IDs, producing the subproblem for the recursion. The mapping itself was mostly applied by assignID

Retrieve the number of suffix arrays.

readInt reads an int x from r using buf to buffer the read and returns x.

recurse_32 calls sais_32 recursively to solve the subproblem we've built. The subproblem is at the right end of sa, the suffix array result will be wr

sais_8_32 computes the suffix array of text. The text must contain only values in [0, textMax). The suffix array is stored in sa, which the caller mus

Same as GenerateGreedy, but will send intermediate results to the generatedTokens

Creates the tokenized corpus and suffix array, saves them to outpath, and returns the model. If either the tokenized corpus or suffix array already ex

Given a sequence of tokens (queryIds) will greedily generate numNewTokens tokens using the longest possible suffix. The suffix must have at least minM

Given a sequence of tokens (queryIds) will print the top-k most likely continuations using the longest possible suffix. The suffix must have at least

assignID_8_32 assigns a dense ID numbering to the set of LMS-substrings respecting string ordering and equality, returning the maximum assigned ID. Fo

Create a suffix array for a given byte array. Each token is a uint16 value, so will take up two bytes. This means that only even indices in the suffix

Reads as many documents as possible (delineated by the sentinal) from filename into slice chunk. Will try to read as many documents as possible into c

Encode a sequence of integers into a byte array ending in the sentinal. The sentinalVal is repeated sentinalSize times.

expand_8_32 distributes the compacted, sorted LMS-suffix indexes from sa[:numLMS] into the tops of the appropriate buckets in sa, preserving the sorte

induceL_8_32 inserts L-type text indexes into sa, assuming that the leftmost S-type indexes are inserted into sa, in sorted order, in the right bucket

induceSubL_8_32 inserts the L-type text indexes of LMS-substrings into sa, assuming that the final characters of the LMS-substrings are already insert

induceSubS_8_32 inserts the S-type text indexes of LMS-substrings into sa, assuming that the leftmost L-type text indexes are already inserted into sa

length_8_32 computes and records the length of each LMS-substring in text. The length of the LMS-substring at index j is stored at sa[j/2], avoiding t

lookupAll returns a slice into the matching region of the index. The runtime is O(log(N)*len(s)).

The SAIS algorithm proceeds in a sequence of scans through sa. Each of the following functions implements one scan, and the functions appear here in t

readSlice reads data[:n] from r and returns n. It uses buf to buffer the read.

Uses binary search to find the occurrences of a query in the suffix array. Returns the starting position of the occurences.

Retrieve the number of occurrences of a query in the suffix array.

Retrieve all occurrences of a query in the suffix array. The returned occurrences are extended by extend tokens.

retrieve number of continuations