GenAI on EKS Starter Kit

A starter kit for deploying and managing GenAI components and examples on Amazon EKS (Elastic Kubernetes Service). This project provides a collection of tools, configurations, components and examples to help you quickly set up a GenAI project on Kubernetes.

The starter kit includes the configurable components and examples from several categories:

• AI Gateway - LiteLLM, Kong AI Gateway OSS
• LLM Model - vLLM, SGLang, Ollama
• Embedding Model - Text Embedding Inference (TEI)
• Observability (o11y) - Langfuse, Phoenix
• GUI App - Open WebUI
• Vector Database - Qdrant, Chroma, Milvus
• Workflow Automation - n8n
• AI Agent - OpenClaw
• MCP Server - FastMCP 2.0
• AI Agent Framework - Strands Agents , Agno

Prerequisites

Before you begin, ensure you have the following tools installed:

• AWS CLI
• Terraform
• Node.js
• kubectl
• Helm
• Docker and Buildx

Initial Setup

1. Install dependencies:

npm install

1. Configure environment variables:

./cli configure
# Example:
# ✔ Enter value for REGION: us-west-2
# ✔ Enter value for EKS_CLUSTER_NAME: genai-on-eks
# ? Enter value for DOMAIN:

This will prompt you to enter values for environment variables. Then, it will save the values on .env.local. There are a few important ones, including:

• REGION - AWS region to be used to provision the infrastructure
• EKS_CLUSTER_NAME - Name of the EKS cluster
• EKS_MODE - EKS mode, auto (default) or standard
• DOMAIN - Recommend to use a domain name already configured with a Route 53 hosted zone, check FAQs more details
• HF_TOKEN - Hugging Face user access token

Environment Setup

There are two methods to setup your environment:

Method 1: Quick Demo Setup

To quickly set up a demo environment with infrastructure and essential components and examples:

./cli demo-setup

This command will:

1. Set up the required infrastructure using Terraform (check Infrastructure Setup for more information)
2. Deploy the demo components and examples specified in the config.json file in the right order

Check Demo Walkthrough on how to setup and use the demo

Method 2: Interactive Setup

For a more customized setup, you can use the interactive setup command:

./cli interactive-setup
# Example:
# ✔ Select AI Gateway components to install: litellm
# ✔ Select LLM Model components to install: vllm
# ? Select Embedding Model components to install: (Press <space> to select, <a> to toggle all, <i> to invert selection, and <enter> to proceed)
# ❯◉ Text Embedding Inference (TEI)

This command will:

1. Present you with a list of available components and examples organized by category
2. Allow you to select which components and examples you want to install
3. Set up the required infrastructure using Terraform
4. Install all the selected components and examples

Note. Unlike the quick demo setup, the selected components and examples may not be deployed in the required order. Some components/examples might need to be refreshed by running the CLI install command again.

NVIDIA Dynamo Platform Setup

This starter kit supports deploying NVIDIA Dynamo for optimized LLM inference on Amazon EKS.

Quick Start

1. Install components in order:

./cli nvidia-platform monitoring install      # Prometheus + Grafana
./cli nvidia-platform gpu-operator install     # NVIDIA GPU Operator
./cli nvidia-platform dynamo-platform install  # Dynamo CRDs, Operator, etcd, NATS
./cli nvidia-platform dynamo-vllm install      # Deploy a model with vLLM

1. Optionally run benchmarks and auto-configuration:

./cli nvidia-platform benchmark install       # AIPerf concurrency sweep
./cli nvidia-platform aiconfigurator install   # TP/PP recommendation + SLA deploy

For full details on platform prerequisites, deployment modes (aggregated vs disaggregated), KV cache routing, monitoring dashboards, benchmarking, and AIConfigurator, see the NVIDIA Platform README.

Components & Examples Management

You can install or uninstall individual components/examples using the CLI:

Install a Component/Example

./cli <category> <component/example> install

# Examples:
# ./cli ai-gateway litellm install
# ./cli strands-agents calculator-agent install

Uninstall a Component/Example

./cli <category> <component/example> uninstall

# Examples:
# ./cli ai-gateway litellm uninstall
# ./cli strands-agents calculator-agent uninstall

LLM/Embedding Model Management

The CLI provides commands to manage LLM/Embedding models for the hosting components:

Configure Models

Configure which models should be deployed for a specific component:

./cli llm-model <component> configure-models
./cli embedding-model <component> configure-models

# Example:
# ./cli llm-model vllm configure-models
# ./cli embedding-model tei configure-models

Update Models

Add and/or remove models for a specific component:

./cli llm-model <component> update-models
./cli embedding-model <component> update-models

# Example:
# ./cli llm-model vllm update-models
# ./cli embedding-model tei update-models

Add Models

Only add missing models for a specific component:

./cli llm-model <component> add-models
./cli embedding-model <component> add-models

# Example:
# ./cli llm-model vllm add-models
# ./cli embedding-model tei add-models

Remove All Models

Remove all models for a specific component:

./cli llm-model <component> remove-all-models
./cli embedding-model <component> remove-all-models

# Example:
# ./cli llm-model vllm remove-all-models
# ./cli embedding-model tei remove-all-models

Cleanup

There are two methods to clean up your environment:

Method 1: Uninstall Components/Examples Individually and Destroy Infrastructure

This method gives you more control over the cleanup process:

1. Uninstall each component/example:

# Examples:
# ./cli strands-agents calculator-agent uninstall
# ./cli ai-gateway litellm uninstall
# ... uninstall other components/examples as needed

1. Destroy the infrastructure:

./cli cleanup-infra

Method 2: Cleanup Everything at Once

This method provides a one-command solution to clean up all examples, components and infrastructure:

./cli cleanup-everything

This command will:

1. Attempt to uninstall all deployed examples and components
2. Destroy the infrastructure using Terraform

FAQs

How the config files work?

.env and config.json will be loaded first. Then, the configs will be merged/overriden with the values from .env.local and config.local.json if exist.

How can I use this starter kit without having a Route 53 hosted zone?

With a domain name already configured with a Route 53 hosted zone, a single shared ALB with HTTPS is used together with a wildcard ACM cert and Route 53 DNS records to expose all public facing services e.g. litellm.

Alternatively, when the DOMAIN filed on .env (or .env.local) is empty, mulitple ALBs with HTTP will be created for each public facing service. In this case, only one service requiring the Nginx Ingress basic auth (e.g. Milvus and Qdrant) can be exposed.

How can I configure and update the LiteLLM proxy model list?

Run ./cli litellm install again to update the LiteLLM models.

For Bedrock models, the model list hardcoded on config.json.

# Example:
"bedrock": {
  "llm": {
    "models": [
      { "name": "amazon-nova-premier", "model": "us.amazon.nova-premier-v1:0" },
      { "name": "claude-4-opus", "model": "us.anthropic.claude-opus-4-20250514-v1:0" },
    ]
  }
}

How can I change the EC2 GPU instance families and purchasing options?

The default instance families are g6e, g6 and g5g and the default purchasing options are spot and on-demand. You can change the values on terraform/0-common.tf and then run ./cli terraform apply again.

Note that the model deployment manifests use nodeSelector like eks.amazonaws.com/instance-family: g6e to lock the specific tested instance family which you will need to adjust accordingly.

For self-hosted models, they will be dynamically detected from the running model pods.

How can I use LLM models with AWS Neuron and EC2 Inferentia 2?

The supported models will have the -neuron suffix. To enable the support, on config.json (or config.local.json), change enableNeuron to true and then install the component again (e.g ./cli llm-model vllm install) which will take ~20-30 mins to build the vLLM Neuron container image and push it to ECR.

When a supported LLM model is deployed for the first time, Neorun just-in-time (JIT) compilation will compile the model which will take ~20-30 mins. The compiled model then will be cached on EFS file system for the subsequent deployments.

Llama-3.1-8B-Instruct, DeepSeek-R1-Distill-Llama-8B, Mistral-7B-Instruct-v0.3 models can the INT8 quantization to run on single inf2.xlarge, but inf2.8xlarge is still required to compile.

• 1st step is to deploy the model with will used inf2.8xlarge first to compile and cache the model
• Then, on config.json (or config.local.json), change from "compile": true to "compile": false
• Then, delete the model deploymen and then deploy again which will use inf2.xlarge

How can I disable the multi-arch container image build?

By default, docker buildx is used to build the multi-arch container images. To disable it, modify the docker section on config.json (or config.local.json) to set "useBuildx": false and arch based on your machine OS arch.

How can I use the different AWS region for Bedrock?

By default, the same region as the EKS cluster will be used. To change it, modify the bedrock section on config.json (or config.local.json) to set region to the preferred region.

How can I provision and manage multiple EKS clusters?

You can change the values of the REGION, EKS_CLUSTER_NAME, and DOMAIN fields on .env (or .env.local). Then, Terraform workspace and kubectl context will automatically use those values when running the related ./cli commands.

What is ECR Pull Through Cache and should I enable it?

ECR Pull Through Cache caches external container images (from Docker Hub, GitHub Container Registry) in your private ECR registry. This avoids rate limits from public registries and keeps images within your AWS infrastructure.

Default: Disabled (enable_ecr_pull_through_cache = false)

Why disabled by default? - Cached images are stored in your private ECR, which incurs storage costs - Public registries (Docker Hub, GHCR) work fine for most use cases since EKS nodes have internet access

When to enable: - You're hitting Docker Hub rate limits (anonymous: 100 pulls/6hrs, authenticated: 200 pulls/6hrs) - You want faster, more reliable pulls from within AWS - Your organization requires images to be stored in private registries

To enable:

Docker Hub and GitHub Container Registry require authentication for ECR pull through cache. You'll need to provide credentials for both registries.

1. Get Docker Hub credentials:
2. Create a Docker Hub account at hub.docker.com
3. Generate an access token at Docker Hub Security Settings → "New Access Token"

4. For more information, see Create and manage access tokens in the Docker documentation

5. Get GitHub credentials:

6. Generate a Personal Access Token (classic) at GitHub Settings → Developer settings → Personal access tokens
7. The token needs read:packages scope to pull from GitHub Container Registry

8. For more information, see Managing your personal access tokens in the GitHub documentation

9. Configure credentials in config.local.json:

// config.local.json
{
  "terraform": {
    "vars": {
      "enable_ecr_pull_through_cache": true,
      "dockerhub_username": "your-dockerhub-username",
      "dockerhub_access_token": "your-dockerhub-access-token",
      "github_username": "your-github-username",
      "github_token": "your-github-personal-access-token"
    }
  }
}

1. Apply the changes:

./cli terraform apply

Important: Keep your credentials in config.local.json (which is gitignored) and never commit them to version control. The credentials are stored securely in AWS Secrets Manager with the ecr-pullthroughcache/ prefix.

Supported registries: - vllm/* → Docker Hub - lmsysorg/* → Docker Hub
- ollama/* → Docker Hub - huggingface/* → GitHub Container Registry

Note: When you terraform destroy, the cache rules are deleted but the cached ECR repos