OptiLLM

🚀 2-10x accuracy improvements on reasoning tasks with zero training

🤗 HuggingFace Space • 📓 Colab Demo • 💬 Discussions

OptiLLM is an OpenAI API-compatible optimizing inference proxy that implements 20+ state-of-the-art techniques to dramatically improve LLM accuracy and performance on reasoning tasks - without requiring any model training or fine-tuning.

It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time. A good example of how to combine such techniques together is the CePO approach from Cerebras.

✨ Key Features

• 🎯 Instant Improvements: 2-10x better accuracy on math, coding, and logical reasoning
• 🔌 Drop-in Replacement: Works with any OpenAI-compatible API endpoint
• 🧠 20+ Optimization Techniques: From simple best-of-N to advanced MCTS and planning
• 📦 Zero Training Required: Just proxy your existing API calls through OptiLLM
• ⚡ Production Ready: Used in production by companies and researchers worldwide
• 🌍 Multi-Provider: Supports OpenAI, Anthropic, Google, Cerebras, and 100+ models via LiteLLM

🚀 Quick Start

Get powerful reasoning improvements in 3 simple steps:

# 1. Install OptiLLM
pip install optillm

# 2. Start the server
export OPENAI_API_KEY="your-key-here"
optillm

# 3. Use with any OpenAI client - just change the model name!

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1")

# Add 'moa-' prefix for Mixture of Agents optimization
response = client.chat.completions.create(
    model="moa-gpt-4o-mini",  # This gives you GPT-4o performance from GPT-4o-mini!
    messages=[{"role": "user", "content": "Solve: If 2x + 3 = 7, what is x?"}]
)

Before OptiLLM: "x = 1" ❌
After OptiLLM: "Let me work through this step by step: 2x + 3 = 7, so 2x = 4, therefore x = 2" ✅

📊 Proven Results

OptiLLM delivers measurable improvements across diverse benchmarks:

Full benchmark results below ⬇️

🏗️ Installation

Using pip

pip install optillm
optillm
2024-10-22 07:45:05,612 - INFO - Loaded plugin: privacy
2024-10-22 07:45:06,293 - INFO - Loaded plugin: memory
2024-10-22 07:45:06,293 - INFO - Starting server with approach: auto

Using docker

docker pull ghcr.io/algorithmicsuperintelligence/optillm:latest
docker run -p 8000:8000 ghcr.io/algorithmicsuperintelligence/optillm:latest
2024-10-22 07:45:05,612 - INFO - Loaded plugin: privacy
2024-10-22 07:45:06,293 - INFO - Loaded plugin: memory
2024-10-22 07:45:06,293 - INFO - Starting server with approach: auto

Available Docker image variants:

• Full image (latest): Includes all dependencies for local inference and plugins
• Proxy-only (latest-proxy): Lightweight image without local inference capabilities
• Offline (latest-offline): Self-contained image with pre-downloaded models (spaCy) for fully offline operation

# Proxy-only (smallest)
docker pull ghcr.io/algorithmicsuperintelligence/optillm:latest-proxy

# Offline (largest, includes pre-downloaded models)
docker pull ghcr.io/algorithmicsuperintelligence/optillm:latest-offline

Install from source

Clone the repository with git and use pip install to setup the dependencies.

git clone https://github.com/algorithmicsuperintelligence/optillm.git
cd optillm
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

🔒 SSL Configuration

OptILLM supports SSL certificate verification configuration for working with self-signed certificates or corporate proxies.

Disable SSL verification (development only):

# Command line
optillm --no-ssl-verify

# Environment variable
export OPTILLM_SSL_VERIFY=false
optillm

Use custom CA certificate:

# Command line
optillm --ssl-cert-path /path/to/ca-bundle.crt

# Environment variable
export OPTILLM_SSL_CERT_PATH=/path/to/ca-bundle.crt
optillm

⚠️ Security Note: Disabling SSL verification is insecure and should only be used in development. For production environments with custom CAs, use --ssl-cert-path instead. See SSL_CONFIGURATION.md for details.

Implemented techniques

Implemented plugins

We support all major LLM providers and models for inference. You need to set the correct environment variable and the proxy will pick the corresponding client.

AZURE_API_VERSION

AZURE_API_BASE | - | | Azure OpenAI (Managed Identity) | AZURE_API_VERSION

AZURE_API_BASE | Login required using az login, see docs for details | | LiteLLM | depends on the model | See docs for details |

You can then run the optillm proxy as follows.

```bash python optillm.py 2024-09-06 07:57:14,191 - INFO - Starting server with approach: auto 2024-09-06 07:57:14,191 - INFO - Server configuration: {'approach': 'auto', 'mcts_simulations': 2, 'mcts_exploration': 0.2, 'mcts_depth': 1, 'best_of_n': 3, 'model': 'gpt-4o-mini', 'rstar_max_depth': 3, 'rstar_num_rollouts': 5, 'rstar_c': 1.4, 'base_url': '', 'host': '127.0.0.1'} * Serving Flask app 'optillm'