MCPcopy Create free account
hub / github.com/BerriAI/litellm

github.com/BerriAI/litellm

Chat with this repo
repository ↗ · DeepWiki ↗ · release v1.110.2 ↗ · + Follow · compare 24 versions
234,415 symbols 1,335,669 edges 13,021 files ⚖ MIT 52,537 documented · 22% ● updated todayv1.109.3 · 2026-11-20★ 63,5761,462 open issues

Browse by type

⨍ Functions 202,479◇ Types & classes 26,341↳ Endpoints 5,595
What it actually does AI analysis from the code graph — generated when you open this
loading…
README

🚅 LiteLLM

LiteLLM AI Gateway

Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.

    <a href="https://render.com/deploy?repo=https://github.com/BerriAI/litellm" target="_blank" rel="nofollow"><img src="https://render.com/images/deploy-to-render-button.svg" alt="Deploy to Render" height="40"></a>
    <a href="https://railway.com/deploy/RhvhdC?referralCode=7mRv9K&utm_medium=integration&utm_source=template&utm_campaign=generic"><img src="https://railway.com/button.svg" alt="Deploy on Railway" height="40"></a>
    <a href="https://console.aws.amazon.com/cloudshell/home" target="_blank" rel="nofollow"><img src="https://github.com/BerriAI/litellm/raw/v1.110.2/github/deploy-on-aws.png" alt="Deploy on AWS" height="40"></a>
    <a href="https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FBerriAI%2Fterraform-google-litellm&cloudshell_git_branch=v1.105.0&cloudshell_workspace=examples%2Fdefault&cloudshell_tutorial=TUTORIAL.md&cloudshell_image=gcr.io/ds-artifacts-cloudshell/deploystack_custom_image&shellonly=true" target="_blank" rel="nofollow"><img src="https://github.com/BerriAI/litellm/raw/v1.110.2/github/deploy-on-gcp.png" alt="Deploy on GCP" height="40"></a>

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website

PyPI Version GitHub Stars Y Combinator W23 Whatsapp Discord Slack CodSpeed

LiteLLM AI Gateway


What is LiteLLM

LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.

Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.

Jump to LiteLLM Proxy (LLM Gateway) Docs

Jump to Supported LLM Providers


Why LiteLLM

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

  • Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
  • Drop-in OpenAI compatibility — swap providers without rewriting your code
  • Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
  • 8ms P95 latency at 1k RPS (benchmarks)

OSS Adopters

Stripe image Google ADK Greptile OpenHands

Netflix

OpenAI Agents SDK

Features

LLMs - Call 100+ LLMs (Python SDK + AI Gateway)

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

uv add litellm

An independent litellm-core distribution provides the Python SDK with the same import litellm API and runtime dependencies. It has no optional extras, CLI entry points, or bundled dashboard. Install one SDK distribution per environment because litellm and litellm-core own overlapping Python files. Use litellm for the proxy, CLI, and optional extras

Build and install core from this checkout while its release integration is pending:

python scripts/build_core_distribution.py --out-dir dist/core
python -m pip install dist/core/litellm_core-*.whl

The builder requires Git, uv and the Rust build toolchain. It reads the release version from the root pyproject.toml, leaves source files unchanged, and produces a wheel and a self-contained sdist in dist/core. Run installation in a fresh environment without litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic  
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

uv tool install 'litellm[proxy]'
litellm --model gpt-4o
import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Auto Routing - Choose a model for each prompt (Python SDK + AI Gateway)

Python SDK

Docs: Autorouter SDK

Install with uv add litellm and set OPENAI_API_KEY for the models below.

import asyncio
from litellm import Router

router = Router(model_list=[
    {"model_name": "fast", "litellm_params": {"model": "openai/gpt-6-luna"}},
    {"model_name": "capable", "litellm_params": {"model": "openai/gpt-6.1-sol"}},
    {"model_name": "auto", "litellm_params": {
        "model": "auto_router/complexity_router",
        "complexity_router_config": {
            "classifier_type": "heuristic",
            "tiers": {"SIMPLE": "fast", "MEDIUM": "fast",
                      "COMPLEX": "capable", "REASONING": "capable"},
            "return_raw_model_name": True,
        },
        "complexity_router_default_model": "capable",
    }},
])
response = asyncio.run(router.acompletion(
    model="auto", messages=[{"role": "user", "content": "What is 2+2?"}],
))
print(response.model, response.choices[0].message.content)

AI Gateway (Proxy Server)

Create an Auto Router, then call its model name. Replace auto and the URL below with yours, and set LITELLM_API_KEY to your virtual key.

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"What is 2+2?"}]}'

A2A - Invoke A2A Agents (Python SDK + AI Gateway)

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent

Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)

import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer <your-master-key>"}    # LiteLLM master key or a virtual key

async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
    resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    config = ClientConfig(
        httpx_client=http_client,
        streaming=False,
        supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
    )
    client = ClientFactory(config).create(agent_card)

    request = SendMessageRequest(
        message=Message(
            message_id=uuid4().hex,
            role=Role.ROLE_USER,
            parts=[Part(text="Hello!")],
        )
    )
    async for event in client.send_message(request):
        populated = event.ListFields()
        if populated and populated[0][0].name in ("message", "msg"):
            print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))

Docs: A2A Agent Gateway

MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)

Python SDK - MCP Bridge

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer <your-master-key>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer <your-master-key>"
      }
    }
  }
}

For MCP OAuth, an upstream may advertise dynamic client registration but refuse requests with HTTP 401 or 403. If the provider requires a pre-registered OAuth app, configure its credentials.client_id and, when required, credentials.client_secret on the MCP server. This skips dynamic registration in the gateway sign-in flow. The provider must approve the app for MCP access; reaching its authorization page does not establish that login or tool calls will succeed

Docs: MCP Gateway

Agents - Run Claude Code, Codex, OpenCode, pi or Deep Agents on any model (Python SDK)

Python SDK - Agents

```python import litellm from litellm import Harness, sandbox

result = litellm.agent( Harness.CLAUDE_CODE, # or Harness.CODEX, Harness.OPENCODE, Harness.PI, Harness.DEEPAGENTS "Find why tests/test_router.py is flaky and fix it.", sandbox=sandbox.local("./repo"), model="litellm_proxy/claude-sonnet-4-5", # a model group on your AI Gateway )

print(re

Extension points exported contracts — how you extend this code

browse all types & interfaces →

Core symbols most depended-on inside this repo

browse all functions →

Shape

Function 136,454
Method 66,025
Class 23,709
Route 5,595
Interface 1,987
Enum 573
Struct 72

Languages

Python84%
TypeScript11%
Rust5%
Go1%
Ruby1%

Modules by API surface

litellm/proxy/swagger/swagger-ui-bundle.js4,347 symbols
litellm/proxy/_experimental/out/_next/static/chunks/085r5ieqhajig.js2,017 symbols
tests/unit/proxy/mcp_server/test_mcp_server_manager.py1,086 symbols
tests/unit/router_strategy/test_complexity_router.py1,032 symbols
tests/unit/test_router/test_router.py956 symbols
litellm/proxy/_experimental/out/_next/static/chunks/2hk8n3azev--d.js902 symbols
tests/unit/proxy/test_common_request_processing.py830 symbols
tests/unit/proxy/management_endpoints/test_key_management_endpoints.py811 symbols
litellm/proxy/proxy_server.py773 symbols
litellm/proxy/_experimental/out/_next/static/chunks/2_9r2jeem0dh8.js740 symbols
litellm/proxy/_experimental/out/_next/static/chunks/1f2vcab0yti03.js712 symbols
tests/unit/litellm_core_utils/test_litellm_logging.py703 symbols

Dependencies from manifests, versioned

github.com/ProtonMail/go-cryptov1.3.0 · 1×
github.com/agext/levenshteinv1.2.2 · 1×
github.com/apparentlymart/go-textseg/v15v15.0.0 · 1×
github.com/fugue-labs/gollemv0.1.0 · 1×

Datastores touched

litellmDatabase · 1 repos
dbDatabase · 1 repos
(mongodb)Database · 1 repos
(mysql)Database · 1 repos
leak_dbDatabase · 1 repos
litellm_dbDatabase · 1 repos
postgresDatabase · 1 repos
unusedDatabase · 1 repos

For agents

$ claude mcp add litellm \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

Ask about this repo answers extend the page