hub / github.com/MiXaiLL76/auto_ai_router
github.com/MiXaiLL76/auto_ai_router @v0.4.2
Chat with this repo
3,041 symbols
13,737 edges
288 files
1,631 documented · 54%
What it actually does AI analysis from the code graph — generated when you open this
loading…
README
Auto AI Router
High-performance proxy router for LLM APIs with automatic load balancing, rate limiting, and fail2ban protection. Routes requests to OpenAI, Vertex AI, Gemini AI Studio, Anthropic, and other Auto AI Router instances.
Key Features
- Multi-provider support — OpenAI, Vertex AI, Gemini, Anthropic, Proxy chains
- Round-robin load balancing — across multiple credentials per model
- Rate limiting — per-credential and per-model RPM/TPM controls
- Fail2ban — automatic provider banning on repeated errors
- Prometheus metrics — request counts, latency, credential status
- LiteLLM DB integration — spend logging and API key authentication
- Streaming — full SSE support for all providers
- Environment variables — secure credential management via
os.environ/VAR_NAME
Quick Start
# Build
git clone https://github.com/MiXaiLL76/auto_ai_router.git
cd auto_ai_router
go build -o auto_ai_router ./cmd/server/
# Run
./auto_ai_router -config config.yaml
Or with Docker:
docker pull ghcr.io/mixaill76/auto_ai_router:latest
docker run -p 8080:8080 -v $(pwd)/config.yaml:/app/config.yaml ghcr.io/mixaill76/auto_ai_router:latest
Documentation
Full documentation is available at mixaill76.github.io/auto_ai_router.
License
Apache License 2.0 — see LICENSE file.
Extension points exported contracts — how you extend this code
Result (Interface)Result represents the outcome of a job execution. [5 implementers]
internal/worker/pool.go
ModelManagerInterface (Interface)
ModelManagerInterface for adding dynamically loaded models [4 implementers]
internal/proxy/remote.go
ProviderResponses (Interface)
ProviderResponses converts between the Responses API format and a specific provider's native request/response format wit [4 …
internal/converter/responses/converter.go
Manager (Interface)
Manager is the main interface for the litellmdb module [3 implementers]
internal/litellmdb/manager.go
Store (Interface)
Store is the pluggable response storage interface. Implementations: bboltStore (local), redisStore (distributed). [3 implementers]
internal/responsestore/store.go
ModelChecker (Interface)
ModelChecker interface for checking model availability [2 implementers]
internal/balancer/roundrobin.go
Job (Interface)
Job represents a unit of work to be processed by a worker. Implementations should define their own concrete job types wi [1 …
internal/worker/pool.go
StreamUsageExtractor (Interface)
StreamUsageExtractor provides a provider-agnostic interface for extracting usage information from streaming response chu [2 …
internal/proxy/stream.go
Core symbols most depended-on inside this repo
Errorcalled by 334
internal/worker/pool.go
Set
called by 204
internal/proxy/sessions.go
Len
called by 191
internal/litellmdb/auth/cache.go
Close
called by 183
internal/responsestore/store.go
New
called by 168
internal/ratelimit/rpm.go
Header
called by 130
internal/proxy/proxy_websocket.go
RecordResponse
called by 112
internal/fail2ban/fail2ban.go
NewTestLogger
called by 100
internal/testhelpers/logger.go
Shape
Languages
Go87%
Python13%
Modules by API surface
internal/ratelimit/rpm_test.go75 symbols
internal/converter/responses/request_test.go56 symbols
internal/balancer/roundrobin_test.go55 symbols
internal/ratelimit/rpm.go51 symbols
internal/models/manager.go45 symbols
tests/test_langchain_tools.py41 symbols
internal/models/manager_test.go41 symbols
internal/litellmdb/manager.go41 symbols
internal/ratelimit/redis_backend_test.go39 symbols
internal/litellmdb/models/models_test.go38 symbols
tests/vertex/test_thinking.py36 symbols
internal/config/config_test.go36 symbols
Datastores touched
dbDatabase · 1 repos
litellmDatabase · 1 repos
(mysql)Database · 1 repos
mydbDatabase · 1 repos
litellmDatabase · 1 repos
nonexistentDatabase · 1 repos
For agents
$ claude mcp add auto_ai_router \
-- python -m otcore.mcp_server <graph>