MCPcopy Create free account
hub / github.com/microsoft/BitNet / Vocab

Class Vocab

utils/convert.py:409–416  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

407
408@runtime_checkable
409class Vocab(BaseVocab, Protocol):
410 vocab_size: int
411 added_tokens_dict: dict[str, int]
412 added_tokens_list: list[str]
413 fname_tokenizer: Path
414
415 def __init__(self, base_path: Path): ...
416 def all_tokens(self) -> Iterable[tuple[bytes, float, gguf.TokenType]]: ...
417
418
419class BpeVocab(Vocab):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected