MCPcopy Index your code
hub / github.com/Doorman11991/smallcode / _tokenize

Method _tokenize

src/knowledge/loader.js:164–169  ·  view source on GitHub ↗

* Tokenize a query into normalized lowercase words ≥ 3 chars.

(text)

Source from the content-addressed store, hash-verified

162 * Tokenize a query into normalized lowercase words ≥ 3 chars.
163 */
164 _tokenize(text) {
165 if (!text || typeof text !== 'string') return [];
166 return text.toLowerCase()
167 .split(/\W+/)
168 .filter(w => w.length >= 3 && !STOP_WORDS.has(w));
169 }
170
171 /**
172 * Score an entry against a query. Higher = more relevant.

Callers 2

_scoreEntryMethod · 0.95
selectForQueryMethod · 0.95

Calls 1

hasMethod · 0.45

Tested by

no test coverage detected