Return 4-tuple of POSIX path strings crafted from the ``purl`` package PURL string or object. The tuple members are: (purl_hash, core_path, purl.version, extra_path) These members can be joined using a POSIX "/" path separator to store package data distributed evenly in many directo
(purl: Union[PackageURL, str])
| 212 | |
| 213 | |
| 214 | def package_path_elements(purl: Union[PackageURL, str]): |
| 215 | """ |
| 216 | Return 4-tuple of POSIX path strings crafted from the ``purl`` package PURL string or object. |
| 217 | The tuple members are: (purl_hash, core_path, purl.version, extra_path) |
| 218 | These members can be joined using a POSIX "/" path separator to store package data distributed |
| 219 | evenly in many directories, where package data of the same package is co-located in the same |
| 220 | root directory. |
| 221 | |
| 222 | The approach is to distribute the files in many directories to avoid having too many data files |
| 223 | in any directory and be able to find the path to the YAML data files for a package given its |
| 224 | PURL. For this we use the first characters of the "purl hash" to construct a path. |
| 225 | |
| 226 | A purl hash has 8,192 possible values, meaning 8,192 directories or repositories, basically used |
| 227 | as a hash table. Given an estimated count of packages of about 30 million in mid 2024, this |
| 228 | gives ample distribution of about 4,000 packages in each of these top level directories and some |
| 229 | room to grow. |
| 230 | |
| 231 | The size to store compressed package metadata is guesstimated to be 1MB on average and 10MB for |
| 232 | a full scan. This means that each directory will store 4K * 10MB ~= 4 GB. This should keep |
| 233 | backing git repositories to a reasonable size, below 5GB. |
| 234 | |
| 235 | The storage scheme is designed to create this path structure: |
| 236 | |
| 237 | <short-purl-hash> : top level directory or repository |
| 238 | <type>/<namespace>/<name> : sub directories |
| 239 | purls.yml : YAML file with known versions for this package ordered from oldest to newest |
| 240 | vulnerabilities.yml : YAML file with known vulnerabilities affecting (and fixed by) this package |
| 241 | |
| 242 | <version> : one sub directory for each version |
| 243 | metadata.yml : ABOUT YAML file with package origin and license metadata for this version |
| 244 | scancode-scan.yml : a scancode scan for this package version |
| 245 | foo-scan.yml : a scan for this package version created with tool foo |
| 246 | sbom.cdx.1.4.json : a CycloneDX SBOM |
| 247 | sbom.cdx.1.5.json : a CycloneDX SBOM |
| 248 | sbom.spdx.2.2.json : a SPDX SBOM |
| 249 | .... other files |
| 250 | |
| 251 | <extra_path> : one sub directory for each quote-encoded <qualifiers#subpath> if any |
| 252 | metadata.yml : ABOUT YAML file with package origin and license metadata for this version |
| 253 | scancode-scan.yml : a scancode scan for this package version |
| 254 | foo-scan.yml : a scan for this package version created with tool foo |
| 255 | sbom.cdx.1.4.json : a CycloneDX SBOM |
| 256 | ... other files |
| 257 | |
| 258 | Some examples: |
| 259 | |
| 260 | We keep the same prefix for different versions:: |
| 261 | |
| 262 | >>> package_path_elements("pkg:pypi/license_expression@30.3.1") |
| 263 | ('50', 'pypi/license-expression', '30.3.1', '') |
| 264 | >>> package_path_elements("pkg:pypi/license_expression@10.3.1") |
| 265 | ('50', 'pypi/license-expression', '10.3.1', '') |
| 266 | |
| 267 | We encode with quotes, avoid double encoding of already quoted parts to make subpaths easier |
| 268 | for filesystems:: |
| 269 | |
| 270 | >>> package_path_elements("pkg:pypi/license_expression@30.3.1?foo=bar&baz=bar#sub/path") |
| 271 | ('50', 'pypi/license-expression', '30.3.1', 'baz%3Dbar%26foo%3Dbar%23sub%2Fpath') |