Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/deshaw/versioned-hdf5
/ functions
Functions
983 in github.com/deshaw/versioned-hdf5
⨍
Functions
983
◇
Types & classes
87
↳
Endpoints
6
↓ 3 callers
Function
format_ndindex
Format a numpy ndindex for pretty-printing. >>> format_ndindex(slice(None)) ':' >>> format_ndindex(slice(None, 10, 2)) ':10:2' >>
versioned_hdf5/tools.py:173
↓ 3 callers
Method
from_dataset
Reverse engineer create_dataset kwargs from an h5py.Dataset.
versioned_hdf5/backend.py:115
↓ 3 callers
Function
hash_one
Hash a single chunk via hash_slab (treating it as a one-chunk slab) and return the 32-byte digest.
tests/test_hash_legacy_compat.py:25
↓ 3 callers
Function
initialize
(f)
versioned_hdf5/backend.py:73
↓ 3 callers
Function
legacy_hash
(h5file)
tests/test_hash_legacy_compat.py:20
↓ 3 callers
Method
len
Length of the first axis.
versioned_hdf5/wrappers.py:1107
↓ 3 callers
Function
normalize_chunks
Normalize the ``chunks`` parameter of create_dataset(), guessing a sensible chunk size when it is not explicitly specified.
versioned_hdf5/backend.py:186
↓ 3 callers
Function
swap
Swap every dataset in old with the corresponding one in new Datasets in old that aren't in new are ignored.
versioned_hdf5/replay.py:816
↓ 3 callers
Method
visititems
(self, func)
versioned_hdf5/wrappers.py:594
↓ 3 callers
Method
whole_chunks_idxidx
(self)
versioned_hdf5/subchunk_map.py:358
↓ 2 callers
Method
__init__
(self, handle: Group)
versioned_hdf5/wrappers.py:68
↓ 2 callers
Method
_add_to_data
(self, name, obj)
versioned_hdf5/wrappers.py:250
↓ 2 callers
Method
_astype_impl
Return self._buffer as a new dtype. Return a view if possible.
versioned_hdf5/wrappers.py:709
↓ 2 callers
Function
_baseline_hash_row
The SHA256 of a chunk, as 4x uint64 (one hash_tables row). Does not cover VLEN dtypes (object strings or NpyStrings).
tests/test_staged_changes.py:1773
↓ 2 callers
Method
_cache_for_handle
Return the wrapper cache of one file handle, creating it if necessary. A cached entry is only reused if it still belongs to this very handle.
versioned_hdf5/wrappers.py:159
↓ 2 callers
Function
_chunk_blocks
Tile `shape` into blocks of whole chunks, each at most `max_bytes` in size. Blocks are grown one axis at a time, starting from the last one, so t
versioned_hdf5/backend.py:578
↓ 2 callers
Method
_committed_version_group
Return a committed version group, wrapped so that it cannot be written to. A read-only file needs no wrapping, as there is nothing to protect
versioned_hdf5/api.py:200
↓ 2 callers
Function
_data_v4_to_sc_hash_table
Load the SHA256 digests from the on-disk hash table and convert it to the layout compatible with `StagedChangesArray.hash_tables`, which is indexe
versioned_hdf5/backend.py:386
↓ 2 callers
Method
_find_all_data_groups
(self)
versioned_hdf5/api.py:373
↓ 2 callers
Method
_find_object_dtype_data_groups
Find all data groups with dtype='O'.
versioned_hdf5/api.py:410
↓ 2 callers
Method
_load_edge_chunks_along_axis
Load the edge chunks along an axis from the base slabs into a new slab
versioned_hdf5/staged_changes.py:1987
↓ 2 callers
Function
_normalize_path
(path: str)
versioned_hdf5/replay.py:796
↓ 2 callers
Method
_pinned_raw_data
Return raw_data for a previously committed dataset, if any. Chunk size and fillvalue are pinned by the first version that commits the
versioned_hdf5/wrappers.py:563
↓ 2 callers
Function
_raw_data_as_base_slab
Return `raw_data`, to be used as a base slab of a StagedChangesArray of the given dtype. Variable-width strings are always stored as object d
versioned_hdf5/backend.py:425
↓ 2 callers
Method
_rebuild_hashtables
Rebuild the hashtables in data_groups.
versioned_hdf5/api.py:381
↓ 2 callers
Function
_recreate_virtual_dataset
Recreate every virtual dataset `name` in the versions `versions` according to the new raw_data chunks in `raw_data_chunks_map`. Returns
versioned_hdf5/replay.py:360
↓ 2 callers
Method
_recursion_to_root
Call `cb` on self and recursively on all parents up to the root. The callable must accept the node being currently visited and the name of the
versioned_hdf5/wrappers.py:545
↓ 2 callers
Function
_rewrite_block_bytes
Scratch-area size, in bytes, for `rewrite_dataset()`. A block of chunks is deduplicated against every chunk already on `raw_data`, and that f
versioned_hdf5/backend.py:613
↓ 2 callers
Method
_set_chunks
(self, dataset_name: str, value: tuple[int, ...] | None)
versioned_hdf5/wrappers.py:582
↓ 2 callers
Method
_untrim_staged_slabs
Deep-copy trimmed staged slabs into new slabs padded to full chunk_size. Trimmed staged slabs are only ever created by from_array(as_base_sla
versioned_hdf5/staged_changes.py:751
↓ 2 callers
Function
_vlen_strings_raw_data
Create the raw_data of a dataset "x" of h5py object strings, 5 points long, with 2 points per chunk.
tests/test_slicetools.py:842
↓ 2 callers
Function
are_compatible_dtypes
Return True if the dtypes are compatible. Compatible dtypes are those that are either equal or both variable length strings.
versioned_hdf5/backend.py:58
↓ 2 callers
Function
basic_idx_st
Hypothesis draw of slice and integer indexes, with potential ellipsis
tests/test_subchunk_map.py:46
↓ 2 callers
Function
buffer
Return a writeable buffer of random data, of shape SHAPE. This mimics the _buffer attribute of an InMemoryArrayDataset, the input to StagedCh
benchmarks/staged_changes.py:142
↓ 2 callers
Method
get_diff
Compute the difference between two versions of a dataset. Parameters ---------- name : str Name of the dataset
versioned_hdf5/api.py:426
↓ 2 callers
Function
get_nth_previous_version
(f, version_name, n)
versioned_hdf5/versions.py:247
↓ 2 callers
Method
get_version_by_name
(self, version)
versioned_hdf5/api.py:216
↓ 2 callers
Function
idx_st
(shape: tuple[int, ...])
tests/test_subchunk_map.py:142
↓ 2 callers
Method
inverse
r""" Return a dictionary mapping Slice: array_of_hash. The Slices are all `reduce()`\d.
versioned_hdf5/hashtable.py:176
↓ 2 callers
Function
ix_with_slices
Variant of numpy.ix_ with added support for mixing with slices. Given a numpy ndindex where each element could be either a 1D fancy integer a
versioned_hdf5/tools.py:111
↓ 2 callers
Function
make_array
Return a NumPy array of the given shape, dtype, memory layout, and sample values.
tests/test_slicetools.py:53
↓ 2 callers
Function
make_committable
Create a StagedChangesArray with staged changes as described by COMMIT_SCENARIOS, ready to be committed
benchmarks/staged_changes.py:237
↓ 2 callers
Method
rand_strings
Generate a ndarray of random strings
benchmarks/common.py:139
↓ 2 callers
Function
read_many_slices_params_nd
Assemble the parameters for read_many_slices() for a multi-dimensional index for a specific transfer type between two arrays. Parameters
versioned_hdf5/subchunk_map.py:863
↓ 2 callers
Function
scalar_indices_st
(size: int)
tests/test_subchunk_map.py:41
↓ 2 callers
Function
set_current_version
(f, version_name)
versioned_hdf5/versions.py:304
↓ 2 callers
Method
setup
(self, kind, chunks)
benchmarks/wrappers.py:147
↓ 2 callers
Function
shape_chunks_st
( draw, max_ndim: int = 4, min_size: int = 1, max_size: int = 20 )
tests/test_subchunk_map.py:131
↓ 2 callers
Function
smallest_step_after
Find the smallest integer y >= x where y = a + k*m for whole k's Assumes 0 <= a <= x and m >= 1. a x y | <-- m --> |
versioned_hdf5/cytools.py:64
↓ 2 callers
Function
stop2count
Given a start:stop:step slice or range, return the number of elements yielded. This is functionally identical to:: len(range(start, stop
versioned_hdf5/cytools.py:17
↓ 2 callers
Method
transfer
Call read_many_slices() to transfer slices of data from src to dst
versioned_hdf5/staged_changes.py:1237
↓ 2 callers
Function
vds_fillvalue
Bake a fillvalue that h5py cannot handle into a VirtualLayout. h5py's ``VirtualLayout.make_dataset()`` stores the fillvalue with ``dcpl.set_f
versioned_hdf5/tools.py:68
↓ 1 callers
Method
__iter__
Iterate over the first axis. BEWARE: Modifications to the yielded data are *NOT* written to file.
versioned_hdf5/wrappers.py:1122
↓ 1 callers
Function
_all_chunks
()
tests/test_hash_legacy_compat.py:42
↓ 1 callers
Function
_all_extant_are_empty
Check if the given versions of a dataset are empty. Doesn't assume the dataset exists in any version. Parameters ---------- f : Vers
versioned_hdf5/replay.py:482
↓ 1 callers
Method
_chunk_sizes_in_chunk_indices
Return the number of points taken from each chunk within self.chunk_indices. All but the last chunk always contain self.chunk_size points.
versioned_hdf5/subchunk_map.py:595
↓ 1 callers
Function
_commit_random
Return a StagedChangesArray covering shape, with all of its chunks on a single base slab complete with hash table, built by staging random data an
benchmarks/staged_changes.py:124
↓ 1 callers
Method
_create_hashtable
(self)
versioned_hdf5/hashtable.py:192
↓ 1 callers
Function
_delete_dataset
Delete the given dataset from the versions.
versioned_hdf5/replay.py:510
↓ 1 callers
Function
_delete_tmp_raw_data
Delete _tmp_raw_data if it exists in the file. Parameters ---------- f : File File in which _tmp_raw_data is to be removed na
versioned_hdf5/replay.py:317
↓ 1 callers
Method
_diff_data
Compute the difference between two versions of a dataset. Parameters ---------- v1 : str Version of a dataset
versioned_hdf5/api.py:503
↓ 1 callers
Method
_enlarge_along_axis
Enlarge along a single axis
versioned_hdf5/staged_changes.py:1771
↓ 1 callers
Function
_exists_in_version
Check if a dataset exists in a given version. Parameters ---------- f : VersionedHDF5File File where the dataset may reside n
versioned_hdf5/replay.py:462
↓ 1 callers
Function
_fancy_idx_to_slices
Break down a one-dimensional integer array into a (greedy) minimal list of slices. Populate input of read_many_slices(). Parameters -----
versioned_hdf5/subchunk_map.py:627
↓ 1 callers
Method
_find_data_groups
Find all groups containing datasets that are descendents of the given node. Parameters ---------- node : h5py.Group
versioned_hdf5/api.py:387
↓ 1 callers
Method
_get_chunks
Get the Dict which maps virtual dataset sources to raw data slices. Parameters ---------- name : str Name of the
versioned_hdf5/api.py:475
↓ 1 callers
Function
_get_np_fillvalue
Get the fillvalue for an empty dataset. See InMemoryDataset.fillvalue. In h5py3 variable length strings use None for the h5py fillvalue, but
versioned_hdf5/replay.py:205
↓ 1 callers
Function
_get_parent
(versions, version_name)
versioned_hdf5/replay.py:634
↓ 1 callers
Method
_get_rand_fn
(cls, dtype)
analysis/generate_data_base.py:101
↓ 1 callers
Function
_index_to_mapper
Convert a one-dimensional index, preprocessed by ndindex, to a mapper
versioned_hdf5/subchunk_map.py:792
↓ 1 callers
Method
_invalidate_all
Forget everything that was read from the file, in every live instance and in every cache, so that it is read again on next access. Th
versioned_hdf5/wrappers.py:171
↓ 1 callers
Method
_invalidate_named
Drop the cached wrappers of the groups at or below ``names``. ``None`` names are ignored, and so are groups that HDF5 can no longer name.
versioned_hdf5/wrappers.py:183
↓ 1 callers
Function
_is_empty
Return True if the dataset at the given version is empty, False otherwise. Assumes the dataset exists in the given version. Parameters -
versioned_hdf5/replay.py:440
↓ 1 callers
Function
_is_under
Whether the group ``name`` is ``prefix`` itself or one of its subgroups.
versioned_hdf5/wrappers.py:90
↓ 1 callers
Method
_load_hashtable
(self, hash_table_dataset)
versioned_hdf5/hashtable.py:209
↓ 1 callers
Method
_modify_dss_dense
( cls, key0_ds, key1_ds, val_ds, num_rows_per_append_0, pct_ch
analysis/generate_data_base.py:370
↓ 1 callers
Method
_modify_dss_dense_old
( cls, key0_ds, key1_ds, val_ds, num_rows_per_append_0, pct_ch
analysis/generate_data.py:351
↓ 1 callers
Method
_modify_dss_sparse
( cls, key0_ds, key1_ds, val_ds, num_rows_per_append, pct_chan
analysis/generate_data_base.py:110
↓ 1 callers
Function
_new_vds_layout
Recreate a VirtualLayout for d, replacing name1 with name2 in the source dset name
versioned_hdf5/replay.py:837
↓ 1 callers
Method
_read_many_slices_param
Return the parameters to read_many_slices() for a single chunk. Returns tuple of: - chunk_sub_start - value_sub_start
versioned_hdf5/subchunk_map.py:272
↓ 1 callers
Method
_relocate_trimmed_staged_chunks_along_axis0
Relocate the edge chunks that lie on staged slabs trimmed along axis 0. Trimmed staged slabs are only ever created by from_array(as_base_slab
versioned_hdf5/staged_changes.py:1855
↓ 1 callers
Function
_replace_attrs_prefix
Replace the prefix name1 with name2 in every string attribute of dataset that holds an absolute path, e.g. the 'raw_data' attribute of a version d
versioned_hdf5/replay.py:807
↓ 1 callers
Method
_repr_idx
Return a string representation of the i-th row
versioned_hdf5/staged_changes.py:1254
↓ 1 callers
Function
_sc_hash_table_to_data_v4
Inverse conversion of `_data_v4_to_sc_hash_table`
versioned_hdf5/backend.py:411
↓ 1 callers
Method
_shrink_along_axis
Shrink along a single axis. Load partial edge chunks into memory to avoid ending up with partially overlapping chunks on disk, e.g. [
versioned_hdf5/staged_changes.py:1743
↓ 1 callers
Function
_transformations
Yield a handful of strided/broadcast views of ``a``.
tests/test_hash.py:523
↓ 1 callers
Method
_visit
(self, prefix, func)
versioned_hdf5/wrappers.py:597
↓ 1 callers
Function
_walk
Traverse the object tree, returning all `raw_data` datasets. We use this instead of version_data.visit(delete_dataset) because visit() has tr
versioned_hdf5/replay.py:540
↓ 1 callers
Method
_write_transactions_dense
( cls, name, chunk_size, compression, versions, print_transact
analysis/generate_data_base.py:347
↓ 1 callers
Method
_write_transactions_dense_old
( cls, name, num_rows_initial_0, num_rows_initial_1, num_transactions,
analysis/generate_data.py:284
↓ 1 callers
Function
action_st
(shape: tuple[int, ...], max_size: int = 20)
tests/test_staged_changes.py:25
↓ 1 callers
Function
array_indices_st
(size: int)
tests/test_subchunk_map.py:34
↓ 1 callers
Method
as_kwargs
Convert to kwargs for create_dataset.
versioned_hdf5/backend.py:110
↓ 1 callers
Function
assert_strings
(actual, expected)
tests/test_replay.py:133
↓ 1 callers
Function
assert_tuple_of_ints
(x: object)
tests/test_staged_changes.py:1196
↓ 1 callers
Function
asv_run
Benchmark whatever is installed in the current environment and store the results under commit_hash
ci/asv_compare.py:84
↓ 1 callers
Function
base_slab
Return (slab, hash_table, slab_indices, slab_offsets) of a single base slab that covers the whole of SHAPE with unique chunks, optionally preceded
benchmarks/staged_changes.py:153
↓ 1 callers
Function
benchmark
(small, large)
tests/test_npystrings.py:247
↓ 1 callers
Function
benchmark
(shape)
tests/test_staged_changes.py:845
↓ 1 callers
Function
bound_slices_st
Hypothesis draw of a slice object to slice an array of <arr_size> points along an axis, returning a view of <view_size> points:: arr = np
tests/test_slicetools.py:236
← previous
next →
101–200 of 983, ranked by callers