Source: https://stackoverflow.com/a/57232670

text_32 returns the suffix array for the input text. It requires that len(text) fit in an int32 and that the caller zero sa.

Tokenize a file from filename using numWorkers processes and writes the resulting tokenized data to outpath. The tokenizer configuration file path is

unmap_8_32 unmaps the subproblem back to the original. sa[:numLMS] is the LMS-substring numbers, which don't matter much anymore. sa[len(sa)-numLMS:]

Writes the even indices into filename with offset added to each value. For writing the suffix array indices to disk: only the even indices align with

writeInt writes an int x to w using buf to buffer the write.

writeSlice writes data[:n] to w and returns n. It uses buf to buffer the write.

Bytes returns the data over which the index was created. It must not be modified.

FindAllIndex returns a sorted list of non-overlapping matches of the regular expression r, where a match is a pair of indices specifying the matched s

Will generate a sequence of numNewTokens tokens greedily using the longest matched suffix. For a suffix to be considered valid, there must be at least

New creates a new [Index] for data. [Index] creation time is O(N) for N = len(data).

Retrieve the number of continuations. Sums over results from each chunk.