MCPcopy Index your code
hub / github.com/InternLM/InternLM / main

Function main

tools/tokenizer.py:129–138  ·  view source on GitHub ↗
()

Source from the content-addressed store, hash-verified

127
128
129def main():
130 # parse arguments
131 args = parse_args()
132
133 text2bin(args.text_input_path, args.bin_output_path)
134 print(f"Successfully converted {args.text_input_path} to {args.bin_output_path}")
135
136 # To avoid potential read/write errors, the metadata preparation follows after creating the .bin file.
137 prepare_meta(args.bin_output_path)
138 print(f"Successfully generated {args.bin_output_path}.meta")
139
140
141if __name__ == "__main__":

Callers 1

tokenizer.pyFile · 0.70

Calls 3

text2binFunction · 0.85
prepare_metaFunction · 0.85
parse_argsFunction · 0.70

Tested by

no test coverage detected