MCPcopy Create free account
hub / github.com/atomicdotdev/atomic / new

Method new

atomic-core/src/diff/token/tokenizer.rs:65–67  ·  view source on GitHub ↗

Create a new tokenizer with default configuration. Uses full code-aware tokenization by default. # Arguments `content` - The bytes to tokenize # Example ```rust use atomic_core::diff::token::Tokenizer; let tokenizer = Tokenizer::new(b"hello world"); let tokens: Vec<_> = tokenizer.collect(); assert_eq!(tokens.len(), 3); ```

(content: &'a [u8])

Source from the content-addressed store, hash-verified

63 /// assert_eq!(tokens.len(), 3);
64 /// ```
65 pub fn new(content: &'a [u8]) -> Self {
66 Self::with_config(content, TokenizerConfig::default())
67 }
68
69 /// Create a new tokenizer with custom configuration.
70 ///

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected