MCPcopy Create free account
hub / github.com/google-research/language / push_examples

Function push_examples

language/realm/preprocessing.py:77–94  ·  view source on GitHub ↗

Create a server and continuously push examples from a generator. Args: example_generator: Generator that yields examples. port: Port for the gRPC server to listen on. max_queue_size: Number of example to queue. queue_timeout: Queue timeout argument. None to block forever, otherwis

(example_generator, port, max_queue_size, queue_timeout)

Source from the content-addressed store, hash-verified

75
76
77def push_examples(example_generator, port, max_queue_size, queue_timeout):
78 """Create a server and continuously push examples from a generator.
79
80 Args:
81 example_generator: Generator that yields examples.
82 port: Port for the gRPC server to listen on.
83 max_queue_size: Number of example to queue.
84 queue_timeout: Queue timeout argument. None to block forever, otherwise time
85 in seconds.
86 """
87 servicer = PreprocessingServicer(max_queue_size=max_queue_size)
88 server = create_and_start_server(port, servicer)
89 for i, example in enumerate(example_generator):
90 servicer.PushExample(example, timeout=queue_timeout)
91 if i % 500 == 0:
92 tf.logging.info("Attempted to push %d examples.", i)
93 tf.logging.info("Queue size is %d.", servicer.QueueSize())
94 server.stop(120.0) # The argument is grace period in seconds.
95
96
97def _make_rpc_op(address, timeout_in_ms):

Callers

nothing calls this directly

Calls 4

PushExampleMethod · 0.95
QueueSizeMethod · 0.95
create_and_start_serverFunction · 0.85

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…