MCPcopy Create free account
hub / github.com/apache/arrow / test_write_dataset_with_backpressure

Function test_write_dataset_with_backpressure

python/pyarrow/tests/test_dataset.py:4348–4420  ·  view source on GitHub ↗
(tempdir)

Source from the content-addressed store, hash-verified

4346@pytest.mark.parquet
4347@pytest.mark.threading
4348def test_write_dataset_with_backpressure(tempdir):
4349 consumer_gate = threading.Event()
4350
4351 # A filesystem that blocks all writes so that we can build
4352 # up backpressure. The writes are released at the end of
4353 # the test.
4354 class GatingFs(ProxyHandler):
4355 def open_output_stream(self, path, metadata):
4356 # Block until the end of the test
4357 consumer_gate.wait()
4358 return self._fs.open_output_stream(path, metadata=metadata)
4359 gating_fs = fs.PyFileSystem(GatingFs(fs.LocalFileSystem()))
4360
4361 schema = pa.schema([pa.field('data', pa.int32())])
4362 # The scanner should queue ~ 8Mi rows (~8 batches) but due to ARROW-16258
4363 # it always queues 32 batches.
4364 batch = pa.record_batch([pa.array(list(range(1_000_000)))], schema=schema)
4365 batches_read = 0
4366 min_backpressure = 32
4367 end = 200
4368 keep_going = True
4369
4370 def counting_generator():
4371 nonlocal batches_read
4372 while batches_read < end:
4373 if not keep_going:
4374 return
4375 time.sleep(0.01)
4376 batches_read += 1
4377 yield batch
4378
4379 scanner = ds.Scanner.from_batches(
4380 counting_generator(), schema=schema, use_threads=True)
4381
4382 write_thread = threading.Thread(
4383 target=lambda: ds.write_dataset(
4384 scanner, str(tempdir), format='parquet', filesystem=gating_fs))
4385 write_thread.start()
4386
4387 try:
4388 start = time.time()
4389
4390 def duration():
4391 return time.time() - start
4392
4393 # This test is timing dependent. There is no signal from the C++
4394 # when backpressure has been hit. We don't know exactly when
4395 # backpressure will be hit because it may take some time for the
4396 # signal to get from the sink to the scanner.
4397 #
4398 # The test may emit false positives on slow systems. It could
4399 # theoretically emit a false negative if the scanner managed to read
4400 # and emit all 200 batches before the backpressure signal had a chance
4401 # to propagate but the 0.01s delay in the generator should make that
4402 # scenario unlikely.
4403 last_value = 0
4404 backpressure_probably_hit = False
4405 while duration() < 10:

Callers

nothing calls this directly

Calls 13

GatingFsClass · 0.85
listFunction · 0.85
counting_generatorFunction · 0.85
PyFileSystemMethod · 0.80
LocalFileSystemMethod · 0.80
timeMethod · 0.80
durationFunction · 0.70
schemaMethod · 0.45
fieldMethod · 0.45
arrayMethod · 0.45
startMethod · 0.45
setMethod · 0.45

Tested by

no test coverage detected