hub / github.com/envoyproxy/ai-gateway

github.com/envoyproxy/ai-gateway @v1.0.0 sqlite

repository ↗ · DeepWiki ↗ · release v1.0.0 ↗

5,510 symbols 23,086 edges 551 files 3,094 documented · 56%

README

Envoy AI Gateway

Envoy AI Gateway is an open source project for using Envoy Gateway to handle request traffic from application clients to Generative AI services.

Usage

When using Envoy AI Gateway, we refer to a two-tier gateway pattern. The Tier One Gateway functions as a centralized entry point, and the Tier Two Gateway handles ingress traffic to a self-hosted model serving cluster.

The Tier One Gateway handles authentication, top-level routing, and global rate limiting
The Tier Two Gateway provides fine-grained control over self-hosted model access, with endpoint picker support for LLM inference optimization.

Supported AI Providers

Envoy AI Gateway supports a wide range of AI providers, making it easy to integrate with your preferred LLM services:

_OpenAI	_{Azure OpenAI}	_{Google Gemini}	_{Vertex AI}	_{AWS Bedrock}
_Mistral	_Cohere	_Groq	_{Together AI}	_DeepInfra
_DeepSeek	_Hunyuan	_SambaNova	_Grok	_{Tetrate Agent Router Service}
_Anthropic

Documentation

Blog introducing Envoy AI Gateway.
Documentation for Envoy AI Gateway.
Quickstart to use Envoy AI Gateway in a few simple steps.
Concepts to understand the architecture and resources of Envoy AI Gateway.
Talks and Presentations about Envoy AI Gateway.

Contact

Slack: Join the Envoy Slack workspace if you're not already a member. Otherwise, use the Envoy AI Gateway channel to start collaborating with the community.

Get Involved

We adhere to the CNCF Code of conduct

The Envoy AI Gateway team and community members meet every Monday. Please register for the meeting, add agenda points, and get involved. The meeting details are available in the public document.

To contribute to the project via pull requests, please read the CONTRIBUTING.md file which includes information on how to build and test the project.

Background

The proposal of using Envoy Gateway as a Cloud Native LLM Gateway inspired the initiation of this project.

Extension points exported contracts — how you extend this code

Spec (Interface)

Spec defines methods for parsing request bodies and selecting translators for different API endpoints. Type Parameters: [10 …

internal/endpointspec/endpointspec.go

BackendAuthHandler (Interface)

BackendAuthHandler is the interface that deals with the backend auth for a specific backend. [8 implementers]

internal/filterapi/runtime.go

Translator (Interface)

Translator translates the request and response messages between the client and the backend API schemas. ReqT represents [19 …

internal/translator/translator.go

Span (Interface)

Span standardizes span interfaces, supporting both streaming and non-streaming endpoints. [9 implementers]

internal/tracing/tracingapi/api.go

BackendSecurityPoliciesGetter (Interface)

BackendSecurityPoliciesGetter has a method to return a BackendSecurityPolicyInterface. A group's client should implement [8 …

api/v1beta1/client/clientset/versioned/typed/api/v1beta1/backendsecuritypolicy.go

BackendSecurityPoliciesGetter (Interface)

BackendSecurityPoliciesGetter has a method to return a BackendSecurityPolicyInterface. A group's client should implement [8 …

api/v1alpha1/client/clientset/versioned/typed/api/v1alpha1/backendsecuritypolicy.go

TokenProvider (Interface)

TokenProvider is an interface for retrieving tokens. [5 implementers]

internal/controller/tokenprovider/token_provider.go

Metrics (Interface)

Metrics is the interface for the base AI Gateway metrics. [2 implementers]

internal/metrics/metrics.go

Core symbols most depended-on inside this repo

Equal

called by 2918

internal/apischema/openai/openai.go

String

called by 2040

internal/testing/ollama.go

Error

called by 444

internal/mcpproxy/handlers.go

InputMessageAttribute

called by 379

internal/tracing/openinference/openinference.go

Get

called by 375

api/v1alpha1/client/listers/api/v1alpha1/mcproute.go

String

called by 354

internal/mcpproxy/session.go

Set

called by 296

internal/tracing/mcp.go

Create

called by 287

api/v1alpha1/client/clientset/versioned/typed/api/v1alpha1/mcproute.go

Shape

Function 2,549

Method 1,686

Struct 1,030

Interface 150

TypeAlias 64

FuncType 19

Route 12

Languages

Go99%

TypeScript1%

Python1%

Modules by API surface

internal/apischema/openai/openai.go494 symbols

api/v1alpha1/zz_generated.deepcopy.go148 symbols

api/v1beta1/zz_generated.deepcopy.go128 symbols

internal/apischema/openai/openai_test.go121 symbols

internal/apischema/anthropic/anthropic.go120 symbols

internal/endpointspec/endpointspec.go66 symbols

internal/tracing/tracer_test.go59 symbols

internal/mcpproxy/handlers_test.go55 symbols

internal/translator/openai_helper.go53 symbols

internal/mcpproxy/handlers.go52 symbols

internal/apischema/anthropic/anthropic_test.go52 symbols

tests/internal/e2elib/e2elib.go51 symbols

Dependencies from manifests, versioned

4d63.com/gocheckcompilerdirectivesv1.3.0 · 1×

4d63.com/gochecknoglobalsv0.2.2 · 1×

al.essio.dev/pkg/shellescapev1.5.1 · 1×

cel.dev/exprv0.25.1 · 1×

cloud.google.com/gov0.121.2 · 1×

cloud.google.com/go/authv0.20.0 · 1×

cloud.google.com/go/auth/oauth2adaptv0.2.8 · 1×

cloud.google.com/go/compute/metadatav0.9.0 · 1×

codeberg.org/chavacava/garifv0.2.0 · 1×

codeberg.org/polyfloyd/go-errorlintv1.9.0 · 1×

dario.cat/mergov1.0.2 · 1×

dev.gaijin.team/go/exhaustruct/v4v4.0.0 · 1×

For agents

$ claude mcp add ai-gateway \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact