Process data sample from input dataset Args: dataset_path (str): Path of dataset json file. sp_model (str): Path of tokenizer. Yields: tuple: dumped processed data sample and length of tokens.
(dataset_path, sp_model)
| 9 | |
| 10 | |
| 11 | def process(dataset_path, sp_model): |
| 12 | """Process data sample from input dataset |
| 13 | |
| 14 | Args: |
| 15 | dataset_path (str): Path of dataset json file. |
| 16 | sp_model (str): Path of tokenizer. |
| 17 | |
| 18 | Yields: |
| 19 | tuple: dumped processed data sample and length of tokens. |
| 20 | """ |
| 21 | |
| 22 | dataset = json.load(open(dataset_path)) |
| 23 | |
| 24 | for data in dataset: |
| 25 | yield tokenize(get_chat_format_data(data), sp_model) |
| 26 | |
| 27 | |
| 28 | def get_chat_format_data(ori_data): |
no test coverage detected