Get the map between dataset split and oss objects. Args: file_map (dict): Structure of data files. e.g., {'train': 'train', 'validation': 'val'}, both of train and val are dirs. objects (list): List of oss objects. e.g., ['train/001/1_123.png', 'train/001/1_124.
(file_map, objects)
| 147 | |
| 148 | |
| 149 | def get_split_objects_map(file_map, objects): |
| 150 | """ |
| 151 | Get the map between dataset split and oss objects. |
| 152 | |
| 153 | Args: |
| 154 | file_map (dict): Structure of data files. e.g., {'train': 'train', 'validation': 'val'}, both of train and val |
| 155 | are dirs. |
| 156 | objects (list): List of oss objects. e.g., ['train/001/1_123.png', 'train/001/1_124.png', 'val/003/3_38.png'] |
| 157 | Returns: |
| 158 | A map of split-objects. e.g., {'train': ['train/001/1_123.png', 'train/001/1_124.png'], |
| 159 | 'validation':['val/003/3_38.png']} |
| 160 | """ |
| 161 | res = {} |
| 162 | for k, v in file_map.items(): |
| 163 | res[k] = [] |
| 164 | |
| 165 | for obj_key in objects: |
| 166 | for k, v in file_map.items(): |
| 167 | if obj_key.startswith(v.rstrip('/') + '/'): |
| 168 | res[k].append(obj_key) |
| 169 | |
| 170 | return res |
| 171 | |
| 172 | |
| 173 | def get_dataset_files(subset_split_into: dict, |
no test coverage detected
searching dependent graphs…