Method all_tokens

utils/convert.py:416–416 · view source on GitHub ↗

(self)

Source from the content-addressed store, hash-verified

414
415	def __init__(self, base_path: Path): ...
416	def all_tokens(self) -> Iterable[tuple[bytes, float, gguf.TokenType]]: ...
417
418
419	class BpeVocab(Vocab):

extract_vocabulary_from_modelMethod · 0.45

no outgoing calls

no test coverage detected