Make dataset and collator for supervised fine-tuning.
(tokenizer: transformers.PreTrainedTokenizer, data_args)
| 401 | return dataset_cls |
| 402 | |
| 403 | def make_supervised_data_module(tokenizer: transformers.PreTrainedTokenizer, data_args) -> Dict: |
| 404 | """Make dataset and collator for supervised fine-tuning.""" |
| 405 | dataset_cls = get_dataset_cls(data_args.dataset_cls) |
| 406 | train_dataset = dataset_cls(tokenizer=tokenizer, data_path=data_args.data_path, data_args=data_args) |
| 407 | data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer) |
| 408 | return dict(train_dataset=train_dataset, eval_dataset=None, data_collator=data_collator) |
no test coverage detected