MCPcopy Create free account
hub / github.com/NVIDIA/NeMo-Relay / wrap_py_llm_stream_exec_fn

Function wrap_py_llm_stream_exec_fn

crates/python/src/py_callable.rs:745–772  ·  view source on GitHub ↗

Wrap a Python async generator `(LlmRequest) -> AsyncIterator[Any]` for LLM stream execution. The returned future resolves to a Rust stream backed by a Tokio task that repeatedly awaits `__anext__()` and forwards JSON-converted chunks through a channel.

(
    py_fn: Py<PyAny>,
)

Source from the content-addressed store, hash-verified

743/// repeatedly awaits `__anext__()` and forwards JSON-converted chunks through a
744/// channel.
745pub fn wrap_py_llm_stream_exec_fn(
746 py_fn: Py<PyAny>,
747) -> Box<
748 dyn Fn(
749 LlmRequest,
750 ) -> Pin<
751 Box<
752 dyn Future<
753 Output = FlowResult<Pin<Box<dyn Stream<Item = FlowResult<Json>> + Send>>>,
754 > + Send,
755 >,
756 > + Send
757 + Sync,
758> {
759 let py_fn = std::sync::Arc::new(py_fn);
760 Box::new(move |request: LlmRequest| {
761 let py_fn = py_fn.clone();
762 Box::pin(async move {
763 let async_iter: Py<PyAny> = Python::attach(|py| {
764 let py_req = PyLLMRequest { inner: request };
765 py_fn
766 .call1(py, (py_req,))
767 .map_err(|e: PyErr| FlowError::Internal(e.to_string()))
768 })?;
769 stream_from_async_iter(async_iter)
770 })
771 })
772}
773
774/// Wrap a Python callable `(Any) -> None` as a collector for streaming LLM calls.
775///

Calls 1

stream_from_async_iterFunction · 0.85