Initialize a loader with configurable options. Note: Loader cannot be reused since it may change inner state when constructing information for loading a graph. Args: source (str or value): The data source to be load, which could be one of the foll
(
self, source, delimiter=",", sep=",", header_row=True, filetype="CSV", **kwargs
)
| 97 | """ |
| 98 | |
| 99 | def __init__( |
| 100 | self, source, delimiter=",", sep=",", header_row=True, filetype="CSV", **kwargs |
| 101 | ): |
| 102 | """Initialize a loader with configurable options. |
| 103 | Note: Loader cannot be reused since it may change inner state when constructing |
| 104 | information for loading a graph. |
| 105 | |
| 106 | Args: |
| 107 | source (str or value): |
| 108 | The data source to be load, which could be one of the followings: |
| 109 | |
| 110 | * local file: specified by URL :code:`file://...` |
| 111 | * oss file: specified by URL :code:`oss://...` |
| 112 | * hdfs file: specified by URL :code:`hdfs://...` |
| 113 | * s3 file: specified by URL :code:`s3://...` |
| 114 | * numpy ndarray, in CSR format |
| 115 | * pandas dataframe |
| 116 | |
| 117 | Ordinary data sources can be loaded using vineyard stream as well, a :code:`vineyard://` |
| 118 | prefix can be used in the URL then the local file, oss object or HDFS file will be loaded |
| 119 | into a vineyard stream first, then GraphScope's fragment will be built upon those streams |
| 120 | in vineyard. |
| 121 | |
| 122 | Once the stream IO in vineyard reaches a stable state, it will be the default mode to |
| 123 | load data sources and construct fragments in GraphScope. |
| 124 | |
| 125 | delimiter (char, optional): Column delimiter. Defaults to ',' |
| 126 | |
| 127 | header_row (bool, optional): Whether source have a header. If true, column names |
| 128 | will be read from the first row of source, else they are named by 'f0', 'f1', .... |
| 129 | Defaults to True. |
| 130 | |
| 131 | filetype (str, optional): Specify the type of files to load, can be "CSV", "ORC", and |
| 132 | "PARQUET". Default is "CSV". |
| 133 | |
| 134 | Notes: |
| 135 | Data is resolved by drivers in `vineyard <https://github.com/v6d-io/v6d>`_ . |
| 136 | See more additional info in `Loading Graph` section of Docs, and implementations in `vineyard`. |
| 137 | """ |
| 138 | self.protocol = "" |
| 139 | # For numpy or pandas, source is the serialized raw bytes |
| 140 | # For files, it's the location |
| 141 | # For vineyard, it's the ID or name |
| 142 | self.source = "" |
| 143 | # options for data source is csv |
| 144 | self.options = CSVOptions() |
| 145 | check_argument( |
| 146 | isinstance(delimiter, str) and len(delimiter) == 1, |
| 147 | "The delimiter must be a single character, cannot be '%s'" % delimiter, |
| 148 | ) |
| 149 | self.options.delimiter = delimiter |
| 150 | self.options.header_row = header_row |
| 151 | self.options.filetype = filetype |
| 152 | # meta for data source is numpy or dataframe |
| 153 | self.deduced_properties = None |
| 154 | # extra args directly passed to storage system |
| 155 | # find more details in fsspec |
| 156 | # https://filesystem-spec.readthedocs.io/en/latest/ |
nothing calls this directly
no test coverage detected