MCPcopy
hub / github.com/togethercomputer/RedPajama-Data / _compute_ngrams

Function _compute_ngrams

app/src/core/document.py:12–13  ·  view source on GitHub ↗
(text_seq, n)

Source from the content-addressed store, hash-verified

10
11
12def _compute_ngrams(text_seq, n):
13 return tuple(form_ngrams(iter(text_seq), n))
14
15
16def split_paragraphs(

Callers 1

__init__Method · 0.85

Calls 1

form_ngramsFunction · 0.90

Tested by

no test coverage detected