MCPcopy Create free account
hub / github.com/atomicdotdev/atomic / minimal

Method minimal

atomic-core/src/diff/token/config.rs:95–103  ·  view source on GitHub ↗

Create a minimal tokenizer config (just words and whitespace). This configuration treats everything as either: - Words (alphanumeric sequences) - Whitespace - Punctuation/Other (single characters) Useful for plain text or when you don't need code-aware tokenization. # Example ```rust use atomic_core::diff::token::{Tokenizer, TokenizerConfig}; let config = TokenizerConfig::minimal(); let token

()

Source from the content-addressed store, hash-verified

93 /// // Numbers and operators won't be specially recognized
94 /// ```
95 pub fn minimal() -> Self {
96 Self {
97 merge_whitespace: true,
98 recognize_operators: false,
99 recognize_strings: false,
100 recognize_numbers: false,
101 recognize_comments: false,
102 }
103 }
104
105 /// Create a code-aware tokenizer config (all features enabled).
106 ///

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected