MCPcopy
hub / github.com/huggingface/datasets / xsplit

Function xsplit

src/datasets/utils/file_utils.py:675–699  ·  view source on GitHub ↗

This function extends os.path.split to support the "::" hop separator. It supports both paths and urls. A shorthand, particularly useful where you have multiple hops, is to “chain” the URLs with the special separator "::". This is used to access files inside a zip file over http for ex

(a)

Source from the content-addressed store, hash-verified

673
674
675def xsplit(a):
676 """
677 This function extends os.path.split to support the "::" hop separator. It supports both paths and urls.
678
679 A shorthand, particularly useful where you have multiple hops, is to “chain” the URLs with the special separator "::".
680 This is used to access files inside a zip file over http for example.
681
682 Let's say you have a zip file at https://host.com/archive.zip, and you want to access the file inside the zip file at /folder1/file.txt.
683 Then you can just chain the url this way:
684
685 zip://folder1/file.txt::https://host.com/archive.zip
686
687 The xsplit function allows you to apply the xsplit on the first path of the chain.
688
689 Example::
690
691 >>> xsplit("zip://folder1/file.txt::https://host.com/archive.zip")
692 ('zip://folder1::https://host.com/archive.zip', 'file.txt')
693 """
694 a, *b = str(a).split("::")
695 if is_local_path(a):
696 return os.path.split(Path(a).as_posix())
697 else:
698 a, tail = posixpath.split(a)
699 return "::".join([a + "//" if a.endswith(":") else a] + b), tail
700
701
702def xsplitext(a):

Callers 1

test_xsplitFunction · 0.90

Calls 2

is_local_pathFunction · 0.85
splitMethod · 0.80

Tested by 1

test_xsplitFunction · 0.72