MCPcopy Create free account
hub / github.com/CarperAI/Code-Pile / Dataset

Class Dataset

codepile/dataset.py:105–132  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

103
104
105class Dataset(ABC):
106 def __init__(self, tempdir, target_dir, *args, **kwargs):
107 self.tempdir = tempdir
108 self.target_dir = target_dir
109
110 self.info : DatasetInfo = None
111
112 self.scraper = None
113 self.processor = None
114 self.analyser = None
115
116 def download(self, *args, **kwargs):
117 self.scraper.scrape()
118
119 def process(self, *args, **kwargs):
120 self.processor.process()
121
122 def analyse(self, *args, **kwargs):
123 self.analyser.analyse()
124
125 '''
126 @property
127 @abstractmethod
128 def info(self) -> DatasetInfo:
129 if self.info is None:
130 raise NotImplementedError()
131 return self.info
132 '''
133
134

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected