MCPcopy Create free account
hub / github.com/atomicdotdev/atomic / with_tokenizer

Method with_tokenizer

atomic-core/src/diff/word.rs:521–524  ·  view source on GitHub ↗

Set the tokenizer configuration.

(mut self, config: TokenizerConfig)

Source from the content-addressed store, hash-verified

519
520 /// Set the tokenizer configuration.
521 pub fn with_tokenizer(mut self, config: TokenizerConfig) -> Self {
522 self.tokenizer_config = config;
523 self
524 }
525}
526
527/// Compute word-level diff between two byte slices.

Callers 1

Calls

no outgoing calls

Tested by 1