Create a minimal tokenizer config (just words and whitespace). This configuration treats everything as either: - Words (alphanumeric sequences) - Whitespace - Punctuation/Other (single characters) Useful for plain text or when you don't need code-aware tokenization. # Example ```rust use atomic_core::diff::token::{Tokenizer, TokenizerConfig}; let config = TokenizerConfig::minimal(); let token
()
| 93 | /// // Numbers and operators won't be specially recognized |
| 94 | /// ``` |
| 95 | pub fn minimal() -> Self { |
| 96 | Self { |
| 97 | merge_whitespace: true, |
| 98 | recognize_operators: false, |
| 99 | recognize_strings: false, |
| 100 | recognize_numbers: false, |
| 101 | recognize_comments: false, |
| 102 | } |
| 103 | } |
| 104 | |
| 105 | /// Create a code-aware tokenizer config (all features enabled). |
| 106 | /// |
nothing calls this directly
no outgoing calls
no test coverage detected