Compute the optimum HDF5 chunksize for I/O purposes. Rational: HDF5 takes the data in bunches of chunksize length to write the on disk. A BTree in memory is used to map structures on disk. The more chunks that are allocated for a dataset the larger the B-tree. Large B-trees take mem
(expected_mb: int)
| 93 | |
| 94 | |
| 95 | def calc_chunksize(expected_mb: int) -> int: |
| 96 | """Compute the optimum HDF5 chunksize for I/O purposes. |
| 97 | |
| 98 | Rational: HDF5 takes the data in bunches of chunksize length to |
| 99 | write the on disk. A BTree in memory is used to map structures on |
| 100 | disk. The more chunks that are allocated for a dataset the larger |
| 101 | the B-tree. Large B-trees take memory and causes file storage |
| 102 | overhead as well as more disk I/O and higher contention for the meta |
| 103 | data cache. You have to balance between memory and I/O overhead |
| 104 | (small B-trees) and time to access to data (big B-trees). |
| 105 | |
| 106 | The tuning of the chunksize parameter affects the performance and |
| 107 | the memory consumed. This is based on my own experiments and, as |
| 108 | always, your mileage may vary. |
| 109 | |
| 110 | """ |
| 111 | expected_mb = limit_es(expected_mb) |
| 112 | zone = int(math.log10(expected_mb)) |
| 113 | expected_mb = 10**zone |
| 114 | chunksize = csformula(expected_mb) |
| 115 | # XXX: Multiply by 8 seems optimal for sequential access |
| 116 | return chunksize * 8 |
| 117 | |
| 118 | |
| 119 | class ChunkInfo(NamedTuple): |
no test coverage detected