MCPcopy Index your code
hub / github.com/huggingface/datasets / assert_arrow_memory_doesnt_increase

Function assert_arrow_memory_doesnt_increase

tests/utils.py:530–536  ·  view source on GitHub ↗
()

Source from the content-addressed store, hash-verified

528
529@contextmanager
530def assert_arrow_memory_doesnt_increase():
531 import gc
532
533 gc.collect()
534 previous_allocated_memory = pa.total_allocated_bytes()
535 yield
536 assert pa.total_allocated_bytes() - previous_allocated_memory <= 0, "Arrow memory wasn't expected to increase."
537
538
539def is_rng_equal(rng1, rng2):

Callers 15

test_dummy_dataset_deepcopyMethod · 0.85
test_dummy_dataset_serializeMethod · 0.85
test_dataset_from_fileFunction · 0.85
test_dataset_from_csv_keep_in_memoryFunction · 0.85
test_dataset_from_json_keep_in_memoryFunction · 0.85
test_dataset_from_parquet_keep_in_memoryFunction · 0.85
test_dataset_from_text_keep_in_memoryFunction · 0.85
test_dataset_from_generator_keep_in_memoryFunction · 0.85
test_dataset_from_sql_keep_in_memoryFunction · 0.85
assert_deepcopy_without_bringing_data_in_memoryFunction · 0.85
assert_pickle_without_bringing_data_in_memoryFunction · 0.85
test_memory_mapped_arrow_table_from_fileFunction · 0.85

Calls

no outgoing calls

Tested by 15

test_dummy_dataset_deepcopyMethod · 0.68
test_dummy_dataset_serializeMethod · 0.68
test_dataset_from_fileFunction · 0.68
test_dataset_from_csv_keep_in_memoryFunction · 0.68
test_dataset_from_json_keep_in_memoryFunction · 0.68
test_dataset_from_parquet_keep_in_memoryFunction · 0.68
test_dataset_from_text_keep_in_memoryFunction · 0.68
test_dataset_from_generator_keep_in_memoryFunction · 0.68
test_dataset_from_sql_keep_in_memoryFunction · 0.68
assert_deepcopy_without_bringing_data_in_memoryFunction · 0.68
assert_pickle_without_bringing_data_in_memoryFunction · 0.68
test_memory_mapped_arrow_table_from_fileFunction · 0.68