Class MPOpenAIAPI

datastudio/models/mp_openai_api.py:84–330

Multiprocessing OpenAI-compatible API using fork + COW shared memory. Distributes requests across multiple worker processes, each running an independent :class:`OpenAIAPI` with its own async event loop. Designed for workloads where single-process async saturates CPU on image encodi

@MODELS.register_module()
class MPOpenAIAPI(BaseAPI):
    """Multiprocessing OpenAI-compatible API using fork + COW shared memory.

    Distributes requests across multiple worker processes, each running an
    independent :class:`OpenAIAPI` with its own async event loop.

    Designed for workloads where single-process async saturates CPU on
    image encoding / JSON serialization, total request count is large,
    and payload size makes pickle-based IPC infeasible.

    Memory model:
        Pre-encoded messages are stored in a module-level global before
        ``fork()``.  Workers inherit the parent's address space via
        Copy-on-Write and only read shared data, so N workers ≈ 1x memory.
    """

    def __init__(
        self,
        model: str,
        key: str = None,
        temperature: float = 0.6,
        top_p: float = 0.95,
        top_k: int = 20,
        min_p: float = 0.0,
        presence_penalty: float = 1.0,
        repetition_penalty: float = 1.0,
        api_base: str = None,
        port: int = None,
        retry: int = 10,
        wait: int = 3,
        timeout: tuple = (30, 1800),
        max_tokens: int = 16384,
        thread_num: int = 8192,
        return_dict: bool = False,
        logger=None,
        max_connections: int = None,
        num_workers: int = None,
        worker_concurrency: int = None,
        enable_thinking: bool = False,
        **kwargs,
    ):
        """Initialize multiprocessing OpenAI API wrapper.

        Args:
            model: Model name, e.g., gpt-4, Qwen2-VL-72B.
            key: API key (uses OPENAI_API_KEY env var if not provided).
            temperature: Generation temperature.
            top_p: Nucleus sampling threshold (0~1).
            top_k: Top-K sampling (0 to disable). vLLM/SGLang extra param.
            min_p: Minimum probability threshold. vLLM/SGLang extra param.
            presence_penalty: Penalize tokens already present in the output.
            repetition_penalty: Penalize repeated tokens. vLLM/SGLang extra param.
            api_base: API base URL (full URL to chat/completions endpoint).
            port: Port number for local deployments.
            retry: Number of retry attempts on API failure.
            wait: Max wait time between retries (seconds).
            timeout: Request timeout as (connect_timeout, read_timeout) tuple.
            max_tokens: Maximum tokens in response.