MCPcopy Create free account
hub / github.com/allenai/molmo2 / get_preprocessor

Function get_preprocessor

tests/data/test_preprocessor.py:12–14  ·  view source on GitHub ↗
()

Source from the content-addressed store, hash-verified

10
11
12def get_preprocessor():
13 tokenizer = build_tokenizer("Qwen/Qwen2-7B")
14 return InterleavedTextPreprocessor(tokenizer=tokenizer)
15
16
17def _test_tokenization(messages, n_at_start=None, preprocessor=None):

Callers 3

_test_tokenizationFunction · 0.85
test_max_tokensFunction · 0.85
test_max_seq_lenFunction · 0.85

Calls 2

build_tokenizerFunction · 0.90
InterleavedTextPreprocessorClass · 0.90

Tested by

no test coverage detected