MCPcopy Create free account
hub / github.com/CarperAI/Code-Pile / Scraper

Class Scraper

codepile/dataset.py:80–87  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

78
79
80class Scraper(ABC):
81 # logic for downloading/scraping the datasets
82 def __init__(self, tempdir, target_dir, *args, **kwargs):
83 self.tempdir = tempdir
84 self.target_dir = target_dir
85
86 def scrape(self):
87 raise NotImplementedError()
88
89
90class Processor(ABC):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected