MCPcopy Create free account
hub / github.com/NVIDIA/NeMo-Relay / wrap_py_llm_request_intercept_fn

Function wrap_py_llm_request_intercept_fn

crates/python/src/py_callable.rs:647–715  ·  view source on GitHub ↗

Wrap a Python callable for unified LLM request intercepts. The Python function receives ``(name: str, request: LlmRequest, annotated: AnnotatedLLMRequest | None)`` and must return ``(LlmRequest, AnnotatedLLMRequest | None)``.

(py_fn: Py<PyAny>)

Source from the content-addressed store, hash-verified

645/// The Python function receives ``(name: str, request: LlmRequest, annotated: AnnotatedLLMRequest | None)``
646/// and must return ``(LlmRequest, AnnotatedLLMRequest | None)``.
647pub fn wrap_py_llm_request_intercept_fn(py_fn: Py<PyAny>) -> LlmRequestInterceptFn {
648 Arc::new(
649 move |name: &str,
650 request: LlmRequest,
651 annotated: Option<AnnotatedLLMRequest>|
652 -> FlowResult<(LlmRequest, Option<AnnotatedLLMRequest>)> {
653 Python::attach(|py| {
654 let py_req = PyLLMRequest {
655 inner: request.clone(),
656 };
657 let py_ann: Py<PyAny> = match annotated {
658 Some(ann) => {
659 let wrapper = PyAnnotatedLLMRequest { inner: ann };
660 wrapper
661 .into_pyobject(py)
662 .map_err(|e| {
663 FlowError::Internal(format!(
664 "Failed to convert AnnotatedLLMRequest to Python: {e}"
665 ))
666 })?
667 .into_any()
668 .unbind()
669 }
670 None => py.None(),
671 };
672 let result = py_fn.call1(py, (name, py_req, py_ann)).map_err(|e| {
673 FlowError::Internal(format!("LLM request intercept callable failed: {e}"))
674 })?;
675
676 // Extract the tuple (LlmRequest, AnnotatedLLMRequest | None)
677 let tuple = result.bind(py);
678 let new_req: PyLLMRequest = tuple
679 .get_item(0)
680 .map_err(|e| {
681 FlowError::Internal(format!(
682 "LLM request intercept result[0] extraction failed: {e}"
683 ))
684 })?
685 .extract()
686 .map_err(|e| {
687 FlowError::Internal(format!(
688 "LLM request intercept result[0] is not LlmRequest: {e}"
689 ))
690 })?;
691 let ann_item = tuple.get_item(1).map_err(|e| {
692 FlowError::Internal(format!(
693 "LLM request intercept result[1] extraction failed: {e}"
694 ))
695 })?;
696 let new_ann = if ann_item.is_none() {
697 None
698 } else {
699 Some(
700 ann_item
701 .extract::<PyAnnotatedLLMRequest>()
702 .map_err(|e| {
703 FlowError::Internal(format!(
704 "LLM request intercept result[1] is not AnnotatedLLMRequest: {e}"

Calls

no outgoing calls