File split_json.py

Megatron-LM/scripts/split_json.py:None–None · view source on GitHub ↗

Source from the content-addressed store, hash-verified

1	"""
2	Takes a corpora of files (specified by `--input_files`) with json data separated
3	by newlines (loose json). Splits data into train.json, val.json, test.json files
4	under `output_dir`.

nothing calls this directly

get_linesFunction · 0.85

get_splitsFunction · 0.85

get_filepathsFunction · 0.85

write_filesFunction · 0.85

appendMethod · 0.80

existsMethod · 0.45

no test coverage detected