Bedrock Access Gateway

OpenAI-compatible RESTful APIs for Amazon Bedrock

What's New 🔥

API Gateway Response Streaming Support - You can now deploy with Amazon API Gateway REST API instead of ALB, enabling true response streaming for better latency and cost optimization. See Deployment Options for details.

Latest Models Supported: - Claude 4.5 Family: Opus 4.5, Sonnet 4.5, Haiku 4.5 - Anthropic's most intelligent models with enhanced coding and agent capabilities - Amazon Nova: Nova Micro, Nova Lite, Nova Pro, Nova Premier - Amazon's native foundation models with multimodal support - DeepSeek: DeepSeek-R1 (reasoning), DeepSeek-V3.1 - Advanced reasoning and general-purpose models - Qwen 3: Qwen3-32B, Qwen3-235B, Qwen3-Coder-30B, Qwen3-Coder-480B - Alibaba's latest language and coding models - OpenAI OSS: gpt-oss-20b, gpt-oss-120b - Open-source GPT models available via Bedrock

It also supports reasoning for Claude 4/4.5 (extended thinking and interleaved thinking) and DeepSeek R1. Check How to Use for more details. You need to first run the Models API to refresh the model list.

Overview

Amazon Bedrock offers a wide range of foundation models (such as Claude 3 Opus/Sonnet/Haiku, Llama 2/3, Mistral/Mixtral, etc.) and a broad set of capabilities for you to build generative AI applications. Check the Amazon Bedrock landing page for additional information.

Sometimes, you might have applications developed using OpenAI APIs or SDKs, and you want to experiment with Amazon Bedrock without modifying your codebase. Or you may simply wish to evaluate the capabilities of these foundation models in tools like AutoGen etc. Well, this repository allows you to access Amazon Bedrock models seamlessly through OpenAI APIs and SDKs, enabling you to test these models without code changes.

If you find this GitHub repository useful, please consider giving it a free star ⭐ to show your appreciation and support for the project.

Features:

• [x] Support streaming response via server-sent events (SSE)
• [x] Support Model APIs
• [x] Support Chat Completion APIs
• [x] Support Tool Call
• [x] Support Embedding API
• [x] Support Multimodal API
• [x] Support Cross-Region Inference
• [x] Support Application Inference Profiles (new)
• [x] Support Reasoning (new)
• [x] Support Interleaved thinking (new)
• [x] Support Prompt Caching (new)

Please check Usage Guide for more details about how to use the new APIs.

Get Started

Prerequisites

Please make sure you have met below prerequisites:

• Access to Amazon Bedrock foundation models.

For more information on how to request model access, please refer to the Amazon Bedrock User Guide (Set Up > Model access)

Architecture

The following diagram illustrates the reference architecture. It uses Amazon API Gateway response streaming with Lambda for SSE support.

Deployment Options

Option	Pros	Cons	Best For
API Gateway + Lambda	No VPC required, pay-per-request, native streaming support, lower operational overhead	Potential cold starts	Most use cases, cost-sensitive deployments
ALB + Fargate	Lowest streaming latency, no cold starts	Higher cost, requires VPC	High-throughput, latency-sensitive workloads

You can also use Lambda Function URL as an alternative, see example

Deployment

Please follow the steps below to deploy the Bedrock Proxy APIs into your AWS account. Only supports regions where Amazon Bedrock is available (such as us-west-2). The deployment will take approximately 10-15 minutes 🕒.

Step 1: Create your own API key in Secrets Manager (MUST)

Note: This step is to use any string (without spaces) you like to create a custom API Key (credential) that will be used to access the proxy API later. This key does not have to match your actual OpenAI key, and you don't need to have an OpenAI API key. please keep the key safe and private.

1. Open the AWS Management Console and navigate to the AWS Secrets Manager service.
2. Click on "Store a new secret" button.
3. In the "Choose secret type" page, select:

Secret type: Other type of secret Key/value pairs: - Key: api_key - Value: Enter your API key value

Click "Next" 4. In the "Configure secret" page: Secret name: Enter a name (e.g., "BedrockProxyAPIKey") Description: (Optional) Add a description of your secret 5. Click "Next" and review all your settings and click "Store"

After creation, you'll see your secret in the Secrets Manager console. Make note of the secret ARN.

Step 2: Build and push container images to ECR

1. Clone this repository: bash git clone https://github.com/aws-samples/bedrock-access-gateway.git cd bedrock-access-gateway

2. Run the build and push script: bash cd scripts bash ./push-to-ecr.sh

3. Follow the prompts to configure:

4. ECR repository names (or use defaults)
5. Image tag (or use default: latest)

6. AWS region (or use default: us-east-1)

7. The script will build and push both Lambda and ECS/Fargate images to your ECR repositories.

8. Important: Copy the image URIs displayed at the end of the script output. You'll need these in the next step.

Step 3: Deploy the CloudFormation stack

1. Download the CloudFormation template you want to use:
2. For API Gateway + Lambda: deployment/BedrockProxy.template

3. For ALB + Fargate: deployment/BedrockProxyFargate.template

4. Sign in to AWS Management Console and navigate to the CloudFormation service in your target region.

5. Click "Create stack" → "With new resources (standard)".

6. Upload the template file you downloaded.

7. On the "Specify stack details" page, provide the following information:

8. Stack name: Enter a stack name (e.g., "BedrockProxyAPI")
9. ApiKeySecretArn: Enter the secret ARN from Step 1
10. ContainerImageUri: Enter the ECR image URI from Step 2 output
11. DefaultModelId: (Optional) Change the default model if needed

Click "Next".

1. On the "Configure stack options" page, you can leave the default settings or customize them according to your needs. Click "Next".

2. On the "Review" page, review all details. Check the "I acknowledge that AWS CloudFormation might create IAM resources" checkbox at the bottom. Click "Submit".

That is it! 🎉 Once deployed, click the CloudFormation stack and go to Outputs tab, you can find the API Base URL from APIBaseUrl, the value should look like http://xxxx.xxx.elb.amazonaws.com/api/v1.

Troubleshooting

If you encounter any issues, please check the Troubleshooting Guide for more details.

SDK/API Usage

All you need is the API Key and the API Base URL. If you didn't set up your own key following Step 1, the application will fail to start with an error message indicating that the API Key is not configured.

Now, you can try out the proxy APIs. Let's say you want to test Claude 3 Sonnet model (model ID: anthropic.claude-3-sonnet-20240229-v1:0)...

Example API Usage

export OPENAI_API_KEY=<API key>
export OPENAI_BASE_URL=<API base url>
# For older versions
# https://github.com/openai/openai-python/issues/624
export OPENAI_API_BASE=<API base url>

curl $OPENAI_BASE_URL/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "anthropic.claude-3-sonnet-20240229-v1:0",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

Example SDK Usage

from openai import OpenAI

client = OpenAI()
completion = client.chat.completions.create(
    model="anthropic.claude-3-sonnet-20240229-v1:0",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(completion.choices[0].message.content)

Please check Usage Guide for more details about how to use embedding API, multimodal API and tool call.

Application Inference Profiles

This proxy now supports Application Inference Profiles, which allow you to track usage and costs for your model invocations. You can use application inference profiles created in your AWS account for cost tracking and monitoring purposes.

Using Application Inference Profiles:

# Use an application inference profile ARN as the model ID
curl $OPENAI_BASE_URL/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/your-profile-id",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

SDK Usage with Application Inference Profiles:

from openai import OpenAI

client = OpenAI()
completion = client.chat.completions.create(
    model="arn:aws:bedrock:us-west-2:123456789012:application-inference-profile/your-profile-id",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(completion.choices[0].message.content)

Benefits of Application Inference Profiles: - Cost Tracking: Track usage and costs for specific applications or use cases - Usage Monitoring: Monitor model invocation metrics through CloudWatch - Tag-based Cost Allocation: Use AWS cost allocation tags for detailed billing analysis

For more information about creating and managing application inference profiles, see the Amazon Bedrock User Guide.

Prompt Caching

This proxy now supports Prompt Caching for Claude and Nova models, which can reduce costs by up to 90% and latency by up to 85% for workloads with repeated prompts.

Supported Models: - Claude models (Claude 3.5 Haiku, Claude 4, Claude 4.5, etc.) - Nova models (Nova Micro, Nova Lite, Nova Pro, Nova Premier)

Enabling Prompt Caching:

You can enable prompt caching in two ways:

1. Globally via Environment Variable (set in ECS Task Definition or Lambda):

ENABLE_PROMPT_CACHING=true

1. Per-request via extra_body :

Python SDK:

from openai import OpenAI

client = OpenAI()

# Cache system prompts
response = client.chat.completions.create(
    model="global.anthropic.claude-haiku-4-5-20251001-v1:0",
    messages=[
        {"role": "system", "content": "You are an expert assistant with knowledge of..."},
        {"role": "user", "content": "Help me with this task"}
    ],
    extra_body={
        "prompt_caching": {"system": True}
    }
)

# Check cache hit
if response.usage.prompt_tokens_details:
    cached_tokens = response.usage.prompt_tokens_details.cached_tokens
    print(f"Cached tokens: {cached_tokens}")

cURL:

curl $OPENAI_BASE_URL/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "global.anthropic.claude-haiku-4-5-20251001-v1:0",
    "messages": [
      {"role": "system", "content": "Long system prompt..."},
      {"role": "user", "content": "Question"}
    ],
    "extra_body": {
      "prompt_caching": {"system": true}
    }
  }'

Cache Options: - "prompt_caching": {"system": true} - Cache system prompts - "prompt_caching": {"messages": true} - Cache user messages - "prompt_caching": {"system": true, "messages": true} - Cache both

Requirements: - Prompt must be ≥1,024 tokens to enable caching - Cache TTL is 5 minutes (resets on each cache hit) - Nova models have a 20,000 token caching limit

For more information, see the Amazon Bedrock Prompt Caching Guide.

Other Examples

LangChain

Make sure you use ChatOpenAI(...) instead of OpenAI(...)

# pip install langchain-openai
import os

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

chat = ChatOpenAI(
    model="anthropic.claude-3-sonnet-20240229-v1:0",
    temperature=0,
    openai_api_key=os.environ['OPENAI_API_KEY'],
    openai_api_base=os.environ['OPENAI_BASE_URL'],
)

template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate.from_template(template)
llm_chain = LLMChain(prompt=prompt, llm=chat)

question = "What NFL team won the Super Bowl in the year Justin Beiber was born?"
response = llm_chain.invoke(question)
print(response)

FAQs

About Privacy

This application does not collect any of your data. Furthermore, it does not log any requests or responses by default.

Why choose API Gateway vs ALB?

API Gateway + Lambda uses API Gateway response streaming with Lambda Web Adapter to support SSE streaming without requiring a VPC. This is a cost-effective, serverless option with up to 10 minutes timeout.

ALB + Fargate provides the lowest streaming latency with no cold starts, ideal for high-throughput workloads.

Which regions are supported?

Generally speaking, all regions that Amazon Bedrock supports will also be supported, if not, please raise an issue in Github.

Note that not all models are available in those regions.

Which models are supported?

You can use the [Model