Index implements a suffix array for fast substring search.

Access the tokenized corpus from a memory-mapped file.

Access the suffix array from a memory-mapped file.

Loads the entire tokenized corpus into memory.

Loads the entire suffix array into memory.

Wrapper around infini-gram model data.

Wrapper around suffix arrays corresponding to multiple chunks of data. Will sum over the results of each chunk.

Wrapper around infini-gram model predictions results.

Wraps the suffix array.

Wraps the tokenized corpus file.

An ints is either an []int32 or an []int64. That is, one of them is empty, and one is the real data. The int64 form is used when len(data) > maxData32