Split the graph into multiple chunks. A directory will be created at :attr:`output_path` with the metadata and chunked edge list as well as the node/edge data. Parameters ---------- g : DGLGraph The graph. name : str The name of the graph, to be used la
(
g, name, ndata_paths, edata_paths, num_chunks, output_path, data_fmt="numpy"
)
| 158 | |
| 159 | |
| 160 | def chunk_graph( |
| 161 | g, name, ndata_paths, edata_paths, num_chunks, output_path, data_fmt="numpy" |
| 162 | ): |
| 163 | """ |
| 164 | Split the graph into multiple chunks. |
| 165 | |
| 166 | A directory will be created at :attr:`output_path` with the metadata and chunked |
| 167 | edge list as well as the node/edge data. |
| 168 | |
| 169 | Parameters |
| 170 | ---------- |
| 171 | g : DGLGraph |
| 172 | The graph. |
| 173 | name : str |
| 174 | The name of the graph, to be used later in DistDGL training. |
| 175 | ndata_paths : dict[str, pathlike] or dict[ntype, dict[str, pathlike]] |
| 176 | The dictionary of paths pointing to the corresponding numpy array file for each |
| 177 | node data key. |
| 178 | edata_paths : dict[etype, pathlike] or dict[etype, dict[str, pathlike]] |
| 179 | The dictionary of paths pointing to the corresponding numpy array file for each |
| 180 | edge data key. ``etype`` could be canonical or non-canonical. |
| 181 | num_chunks : int |
| 182 | The number of chunks |
| 183 | output_path : pathlike |
| 184 | The output directory saving the chunked graph. |
| 185 | """ |
| 186 | for ntype, ndata in ndata_paths.items(): |
| 187 | for key in ndata.keys(): |
| 188 | ndata[key] = os.path.abspath(ndata[key]) |
| 189 | for etype, edata in edata_paths.items(): |
| 190 | for key in edata.keys(): |
| 191 | edata[key] = os.path.abspath(edata[key]) |
| 192 | with setdir(output_path): |
| 193 | _chunk_graph( |
| 194 | g, name, ndata_paths, edata_paths, num_chunks, output_path, data_fmt |
| 195 | ) |
| 196 | |
| 197 | |
| 198 | if __name__ == "__main__": |
no test coverage detected