MCPcopy Create free account
hub / github.com/Pints-AI/1.5-Pints / write_vocab_only

Method write_vocab_only

tokenizer/convert/convert.py:1084–1099  ·  view source on GitHub ↗
(
        fname_out: Path, params: Params, vocab: Vocab, svocab: gguf.SpecialVocab,
        endianess: gguf.GGUFEndian = gguf.GGUFEndian.LITTLE, pad_vocab: bool = False,
    )

Source from the content-addressed store, hash-verified

1082
1083 @staticmethod
1084 def write_vocab_only(
1085 fname_out: Path, params: Params, vocab: Vocab, svocab: gguf.SpecialVocab,
1086 endianess: gguf.GGUFEndian = gguf.GGUFEndian.LITTLE, pad_vocab: bool = False,
1087 ) -> None:
1088 check_vocab_size(params, vocab, pad_vocab = pad_vocab)
1089
1090 of = OutputFile(fname_out, endianess=endianess)
1091
1092 # meta data
1093 of.add_meta_arch(params)
1094 of.add_meta_vocab(vocab)
1095 of.add_meta_special_vocab(svocab)
1096
1097 of.write_meta()
1098
1099 of.close()
1100
1101 @staticmethod
1102 def do_item(item: tuple[str, LazyTensor]) -> tuple[DataType, NDArray]:

Callers 1

mainFunction · 0.80

Calls 7

add_meta_archMethod · 0.95
add_meta_vocabMethod · 0.95
write_metaMethod · 0.95
closeMethod · 0.95
check_vocab_sizeFunction · 0.85
OutputFileClass · 0.85

Tested by

no test coverage detected