MCPcopy Create free account
hub / github.com/OpenSparseLLMs/Linear-MoE / preprocess_wudao2.py

File preprocess_wudao2.py

toolkits/pretrain_data_preprocessing/preprocess_wudao2.py:None–None  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1import argparse
2import codecs
3import json
4import multiprocessing

Callers

nothing calls this directly

Calls 1

mainFunction · 0.70

Tested by

no test coverage detected