MCPcopy Create free account
hub / github.com/atomicdotdev/atomic / tokenize_for_fts

Function tokenize_for_fts

atomic-core/src/pristine/tables.rs:1010–1016  ·  view source on GitHub ↗

Tokenize text for the FTS inverted index. Extracts lowercase alphanumeric words of length >= 3, filtering out stop words (common English + programming terms) that match too many nodes to be useful for ranking.

(text: &str)

Source from the content-addressed store, hash-verified

1008/// stop words (common English + programming terms) that match too many
1009/// nodes to be useful for ranking.
1010pub fn tokenize_for_fts(text: &str) -> Vec<String> {
1011 text.split(|c: char| !c.is_alphanumeric() && c != '_')
1012 .filter(|w| w.len() >= 3)
1013 .map(|w| w.to_lowercase())
1014 .filter(|w| !FTS_STOP_WORDS.contains(&w.as_str()))
1015 .collect()
1016}
1017
1018/// Encode an embedding key as "path\0chunk_idx".
1019#[inline]

Callers 11

search_termsFunction · 0.85
test_tokenize_for_ftsFunction · 0.85
kg_fts_searchMethod · 0.85
kg_fts_match_idsMethod · 0.85
ensure_kg_fts_schemaMethod · 0.85
add_kg_fts_entriesMethod · 0.85
kg_fts_searchMethod · 0.85
kg_fts_match_idsMethod · 0.85
fts_hitsFunction · 0.85
search_termsFunction · 0.85

Calls 4

collectMethod · 0.45
lenMethod · 0.45
containsMethod · 0.45
as_strMethod · 0.45

Tested by 2

test_tokenize_for_ftsFunction · 0.68