Function get_tokenizer

python/tokenizer_util.py:15–38 · view source on GitHub ↗

(tokenizer_name: str, n_vocab: int)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

inference_example.pyFile · 0.90

measure_pexplexity.pyFile · 0.90

generate_completions.pyFile · 0.90

chat_with_bot.pyFile · 0.90

encodeMethod · 0.80

no test coverage detected