MCPcopy Create free account
hub / github.com/AlexWan0/infini-gram / documentIter

Function documentIter

files.go:186–260  ·  view source on GitHub ↗

Reads as many documents as possible (delineated by the sentinal) from filename into slice chunk. Will try to read as many documents as possible into chunk, then call callback with the length of the values read. The function callback is called everytime the chunk is full.

(filename string, sentinalSize, sentinalValue int, chunk []byte, callback func(int) error)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 1

InitializeModelFunction · 0.85

Calls 3

findLastSentinalFunction · 0.85
hasSentinalFunction · 0.85
ReadMethod · 0.80

Tested by

no test coverage detected