MCPcopy
hub / github.com/huggingface/datasets / from_directory

Method from_directory

src/datasets/info.py:248–279  ·  view source on GitHub ↗

Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`. This function updates all the dynamically generated fields (num_examples, hash, time of creation,...) of the [`DatasetInfo`]. This will overwrite all previous metadata. Args: dataset_info_

(cls, dataset_info_dir: str, storage_options: Optional[dict] = None)

Source from the content-addressed store, hash-verified

246
247 @classmethod
248 def from_directory(cls, dataset_info_dir: str, storage_options: Optional[dict] = None) -> "DatasetInfo":
249 """Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`.
250
251 This function updates all the dynamically generated fields (num_examples,
252 hash, time of creation,...) of the [`DatasetInfo`].
253
254 This will overwrite all previous metadata.
255
256 Args:
257 dataset_info_dir (`str`):
258 The directory containing the metadata file. This
259 should be the root directory of a specific dataset version.
260 storage_options (`dict`, *optional*):
261 Key/value pairs to be passed on to the file-system backend, if any.
262
263 <Added version="2.9.0"/>
264
265 Example:
266
267 ```py
268 >>> from datasets import DatasetInfo
269 >>> ds_info = DatasetInfo.from_directory("/path/to/directory/")
270 ```
271 """
272 fs: fsspec.AbstractFileSystem
273 fs, *_ = url_to_fs(dataset_info_dir, **(storage_options or {}))
274 logger.debug(f"Loading Dataset info from {dataset_info_dir}")
275 if not dataset_info_dir:
276 raise ValueError("Calling DatasetInfo.from_directory() with undefined dataset_info_dir.")
277 with fs.open(posixpath.join(dataset_info_dir, config.DATASET_INFO_FILENAME), "r", encoding="utf-8") as f:
278 dataset_info_dict = json.load(f)
279 return cls.from_dict(dataset_info_dict)
280
281 @classmethod
282 def from_dict(cls, dataset_info_dict: dict) -> "DatasetInfo":

Callers 6

__init__Method · 0.45
_load_infoMethod · 0.45
test_from_dirFunction · 0.45
test_test_commandFunction · 0.45

Calls 3

openMethod · 0.80
loadMethod · 0.45
from_dictMethod · 0.45

Tested by 4

test_from_dirFunction · 0.36
test_test_commandFunction · 0.36