Downloads and returns the task dataset. Override this method to download the dataset from a custom API. :param data_dir: str Stores the path to a local folder containing the `Task`'s data files. Use this to specify the path to manually downloaded data (usuall
(self, data_dir=None, cache_dir=None, download_mode=None)
| 221 | self._filters = [build_filter_ensemble("none", [["take_first", None]])] |
| 222 | |
| 223 | def download(self, data_dir=None, cache_dir=None, download_mode=None) -> None: |
| 224 | """Downloads and returns the task dataset. |
| 225 | Override this method to download the dataset from a custom API. |
| 226 | |
| 227 | :param data_dir: str |
| 228 | Stores the path to a local folder containing the `Task`'s data files. |
| 229 | Use this to specify the path to manually downloaded data (usually when |
| 230 | the dataset is not publicly accessible). |
| 231 | :param cache_dir: str |
| 232 | The directory to read/write the `Task` dataset. This follows the |
| 233 | HuggingFace `datasets` API with the default cache directory located at: |
| 234 | `~/.cache/huggingface/datasets` |
| 235 | NOTE: You can change the cache location globally for a given process |
| 236 | by setting the shell environment variable, `HF_DATASETS_CACHE`, |
| 237 | to another directory: |
| 238 | `export HF_DATASETS_CACHE="/path/to/another/directory"` |
| 239 | :param download_mode: datasets.DownloadMode |
| 240 | How to treat pre-existing `Task` downloads and data. |
| 241 | - `datasets.DownloadMode.REUSE_DATASET_IF_EXISTS` |
| 242 | Reuse download and reuse dataset. |
| 243 | - `datasets.DownloadMode.REUSE_CACHE_IF_EXISTS` |
| 244 | Reuse download with fresh dataset. |
| 245 | - `datasets.DownloadMode.FORCE_REDOWNLOAD` |
| 246 | Fresh download and fresh dataset. |
| 247 | """ |
| 248 | self.dataset = datasets.load_dataset( |
| 249 | path=self.DATASET_PATH, |
| 250 | name=self.DATASET_NAME, |
| 251 | data_dir=data_dir, |
| 252 | cache_dir=cache_dir, |
| 253 | download_mode=download_mode, |
| 254 | ) |
| 255 | |
| 256 | @property |
| 257 | def config(self): |
no outgoing calls
no test coverage detected