Method
__init__
(self, tokenizer=None, max_len=2048, test=False, category="", dedup=False, seed=None)
Source from the content-addressed store, hash-verified
| 36 | |
| 37 | class BaseDataset(Dataset): |
| 38 | def __init__(self, tokenizer=None, max_len=2048, test=False, category="", dedup=False, seed=None): |
| 39 | super().__init__() |
| 40 | self.data = None |
| 41 | self.inputs = None |
| 42 | |
| 43 | if tokenizer is not None: |
| 44 | self.tokenizer = Tokenizer(tokenizer) |
| 45 | if seed is not None: |
| 46 | random.seed(seed) |
| 47 | |
| 48 | self.test = test |
| 49 | self.max_len = max_len |
| 50 | self.category = category |
| 51 | self.dedup = dedup |
| 52 | |
| 53 | def __len__(self): |
| 54 | return len(self.data) |
Callers
nothing calls this directly
Tested by
no test coverage detected