MCPcopy Index your code
hub / github.com/microsoft/BitNet / write_vocab_only

Method write_vocab_only

utils/convert.py:1214–1229  ·  view source on GitHub ↗
(
        fname_out: Path, params: Params, vocab: Vocab, svocab: gguf.SpecialVocab,
        endianess: gguf.GGUFEndian = gguf.GGUFEndian.LITTLE, pad_vocab: bool = False,
    )

Source from the content-addressed store, hash-verified

1212
1213 @staticmethod
1214 def write_vocab_only(
1215 fname_out: Path, params: Params, vocab: Vocab, svocab: gguf.SpecialVocab,
1216 endianess: gguf.GGUFEndian = gguf.GGUFEndian.LITTLE, pad_vocab: bool = False,
1217 ) -> None:
1218 check_vocab_size(params, vocab, pad_vocab=pad_vocab)
1219
1220 of = OutputFile(fname_out, endianess=endianess)
1221
1222 # meta data
1223 of.add_meta_arch(params)
1224 of.add_meta_vocab(vocab)
1225 of.add_meta_special_vocab(svocab)
1226
1227 of.write_meta()
1228
1229 of.close()
1230
1231 @staticmethod
1232 def do_item(item: tuple[str, LazyTensor]) -> tuple[DataType, NDArray]:

Callers 1

mainFunction · 0.45

Calls 7

add_meta_archMethod · 0.95
add_meta_vocabMethod · 0.95
write_metaMethod · 0.95
closeMethod · 0.95
check_vocab_sizeFunction · 0.70
OutputFileClass · 0.70

Tested by

no test coverage detected