AI Town 🏠💻💌

Live Demo

Join our community Discord: AI Stack Devs

AI Town is a virtual town where AI characters live, chat and socialize.

This project is a deployable starter kit for easily building and customizing your own version of AI town. Inspired by the research paper Generative Agents: Interactive Simulacra of Human Behavior.

The primary goal of this project, beyond just being a lot of fun to work on, is to provide a platform with a strong foundation that is meant to be extended. The back-end natively supports shared global state, transactions, and a simulation engine and should be suitable from everything from a simple project to play around with to a scalable, multi-player game. A secondary goal is to make a JS/TS framework available as most simulators in this space (including the original paper above) are written in Python.

Overview

• 💻 Stack
• 🧠 Installation (cloud, local, Docker, self-host, Fly.io, ...)
• 💻️ Windows Pre-requisites
• 🤖 Configure your LLM of choice (Ollama, OpenAI, Together.ai, ...)
• 👤 Customize - YOUR OWN simulated world
• 👩‍💻 Deploying to production
• 🐛 Troubleshooting

Stack

• Game engine, database, and vector search: Convex
• Auth (Optional): Clerk
• Default chat model is llama3 and embeddings with mxbai-embed-large.
• Local inference: Ollama
• Configurable for other cloud LLMs: Together.ai or anything that speaks the OpenAI API. PRs welcome to add more cloud provider support.
• Background Music Generation: Replicate using MusicGen

Other credits:

• Pixel Art Generation: Replicate, Fal.ai
• All interactions, background music and rendering on the component in the project are powered by PixiJS.
• Tilesheet:
• https://opengameart.org/content/16x16-game-assets by George Bailey
• https://opengameart.org/content/16x16-rpg-tileset by hilau
• We used https://github.com/pierpo/phaser3-simple-rpg for the original POC of this project. We have since re-wrote the whole app, but appreciated the easy starting point
• Original assets by ansimuz
• The UI is based on original assets by Mounir Tohami

Installation

The overall steps are:

1. Build and deploy
2. Connect it to an LLM

Build and Deploy

There are a few ways to run the app on top of Convex (the backend).

1. The standard Convex setup, where you develop locally or in the cloud. This requires a Convex account(free). This is the easiest way to depoy it to the cloud and seriously develop.
2. If you want to try it out without an account and you're okay with Docker, the Docker Compose setup is nice and self-contained.
3. There's a community fork of this project offering a one-click install on Pinokio for anyone interested in running but not modifying it 😎.
4. You can also deploy it to Fly.io. See ./fly for instructions.

Standard Setup

Note, if you're on Windows, see below.

git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install

This will require logging into your Convex account, if you haven't already.

To run it:

npm run dev

You can now visit http://localhost:5173.

If you'd rather run the frontend and backend separately (which syncs your backend functions as they're saved), you can run these in two terminals:

npm run dev:frontend
npm run dev:backend

See package.json for details.

Using Docker Compose with self-hosted Convex

You can also run the Convex backend with the self-hosted Docker container. Here we'll set it up to run the frontend, backend, and dashboard all via docker compose.

docker compose up --build -d

The container will keep running in the background if you pass -d. After you've done it once, you can stop and start services.

• The frontend will be running on http://localhost:5173.
• The backend will be running on http://localhost:3210 (3211 for the http api).
• The dashboard will be running on http://localhost:6791.

To log into the dashboard and deploy from the convex CLI, you will need to generate an admin key.

docker compose exec backend ./generate_admin_key.sh

Add it to your .env.local file. Note: If you run down and up, you'll have to generate the key again and update the .env.local file.

# in .env.local
CONVEX_SELF_HOSTED_ADMIN_KEY="<admin-key>" # Ensure there are quotes around it
CONVEX_SELF_HOSTED_URL="http://127.0.0.1:3210"

Then set up the Convex backend (one time):

npm run predev

To continuously deploy new code to the backend and print logs:

npm run dev:backend

To see the dashboard, visit http://localhost:6791 and provide the admin key you generated earlier.

Configuring Docker for Ollama

If you'll be using Ollama for local inference, you'll need to configure Docker to connect to it.

npx convex env set OLLAMA_HOST http://host.docker.internal:11434

To test the connection (after you have it running):

docker compose exec backend /bin/bash curl http://host.docker.internal:11434

If it says "Ollama is running", it's good! Otherwise, check out the Troubleshooting section.

Connect an LLM

Note: If you want to run the backend in the cloud, you can either use a cloud-based LLM API, like OpenAI or Together.ai or you can proxy the traffic from the cloud to your local Ollama. See below for instructions.

Ollama (default)

By default, the app tries to use Ollama to run it entirely locally.

1. Download and install Ollama.
2. Open the app or run ollama serve in a terminal. ollama serve will warn you if the app is already running.
3. Run ollama pull llama3 to have it download llama3.
4. Test it out with ollama run llama3.

Ollama model options can be found here.

If you want to customize which model to use, adjust convex/util/llm.ts or set npx convex env set OLLAMA_MODEL # model. If you want to edit the embedding model:

1. Change the OLLAMA_EMBEDDING_DIMENSION in convex/util/llm.ts and ensure: export const EMBEDDING_DIMENSION = OLLAMA_EMBEDDING_DIMENSION;
2. Set npx convex env set OLLAMA_EMBEDDING_MODEL # model.

Note: You might want to set NUM_MEMORIES_TO_SEARCH to 1 in constants.ts, to reduce the size of conversation prompts, if you see slowness.

OpenAI

To use OpenAI, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = OPENAI_EMBEDDING_DIMENSION;

Set the OPENAI_API_KEY environment variable. Visit https://platform.openai.com/account/api-keys if you don't have one.

npx convex env set OPENAI_API_KEY 'your-key'

Optional: choose models with OPENAI_CHAT_MODEL and OPENAI_EMBEDDING_MODEL.

Together.ai

To use Together.ai, you need to:

// In convex/util/llm.ts change the following line:
export const EMBEDDING_DIMENSION = TOGETHER_EMBEDDING_DIMENSION;

Set the TOGETHER_API_KEY environment variable. Visit https://api.together.xyz/settings/api-keys if you don't have one.

npx convex env set TOGETHER_API_KEY 'your-key'

Optional: choose models via TOGETHER_CHAT_MODEL, TOGETHER_EMBEDDING_MODEL. The embedding model's dimension must match EMBEDDING_DIMENSION.

Other OpenAI-compatible API

You can use any OpenAI-compatible API, such as Anthropic, Groq, or Azure.

• Change the EMBEDDING_DIMENSION in convex/util/llm.ts to match the dimension of your embedding model.
• Edit getLLMConfig in llm.ts or set environment variables:

npx convex env set LLM_API_URL 'your-url'
npx convex env set LLM_API_KEY 'your-key'
npx convex env set LLM_MODEL 'your-chat-model'
npx convex env set LLM_EMBEDDING_MODEL 'your-embedding-model'

Note: if LLM_API_KEY is not required, don't set it.

Note on changing the LLM provider or embedding model:

If you change the LLM provider or embedding model, you should delete your data and start over. The embeddings used for memory are based on the embedding model you choose, and the dimension of the vector database must match the embedding model's dimension. See below for how to do that.

Customize your own simulation

NOTE: every time you change character data, you should re-run npx convex run testing:wipeAllTables and then npm run dev to re-upload everything to Convex. This is because character data is sent to Convex on the initial load. However, beware that npx convex run testing:wipeAllTables WILL wipe all of your data.

1. Create your own characters and stories: All characters and stories, as well as their spritesheet references are stored in characters.ts. You can start by changing character descriptions.

2. Updating spritesheets: in data/characters.ts, you will see this code:

ts export const characters = [ { name: 'f1', textureUrl: '/assets/32x32folk.png', spritesheetData: f1SpritesheetData, speed: 0.1, }, ... ];

You should find a sprite sheet for your character, and define sprite motion / assets in the corresponding file (in the above example, f1SpritesheetData was defined in f1.ts)

1. Update the Background (Environment): The map gets loaded in convex/init.ts from data/gentle.js. To update the map, follow these steps:

2. Use Tiled to export tilemaps as a JSON file (2 layers named bgtiles and objmap)

3. Use the convertMap.js script to convert the JSON to a format that the engine can use.

console node data/convertMap.js <mapDataPath> <assetPath> <tilesetpxw> <tilesetpxh>

• <mapDataPath>: Path to the Tiled JSON file.
• <assetPath>: Path to tileset images.
• <tilesetpxw>: Tileset width in pixels.

• <tilesetpxh>: Tileset height in pixels. Generates converted-map.js that you can use like gentle.js

• Adding background music with Replicate (Optional)

For Daily background music generation, create a Replicate account and create a token in your Profile's API Token page. npx convex env set REPLICATE_API_TOKEN # token

This only works if you can receive the webhook from Replicate. If it's running in the normal Convex cloud, it will work by default. If you're self-hosting, you'll need to configure it to hit your app's url on /http. If you're using Docker Compose, it will be http://localhost:3211, but you'll need to proxy the traffic to your local machine.

Note: The simulation will pause after 5 minutes if the window is idle. Loading the page will unpause it. You can also manually freeze & unfreeze the world with a button in the UI. If you want to run the world without the browser, you can comment-out the "stop inactive worlds" cron in convex/crons.ts.

• Change the background music by modifying the prompt in convex/music.ts
• Change how often to generate new music at convex/crons.ts by modifying the generate new background music job

Commands to run / test / debug

To stop the back end, in case of too much activity

This will stop running the engine and agents. You can still run queries and run functions to debug.

npx convex run testing:stop

To restart the back end after stopping it

npx convex run testing:resume

To kick the engine in case the game engine or agents aren't running

npx convex run testing:kick

To archive the world

If you'd like to reset the world and start from scratch, you can archive the current world:

npx convex run testing:archive

Then, you can still look at the world's data in the dashboard, but the engine and agents will no longer run.

You can then create a fresh world with init.

npx convex run init

To pause your backend deployment

You can go to the dashboard to your deployment settings to pause and un-pause your deployment. This will stop all functions, whether invoked from the client, scheduled, or as a cron job. See this as a last resort, as there are gentler ways of stopping above.

Windows Installation

Prerequisites

1. Windows 10/11 with WSL2 installed
2. Internet connection

Steps:

1. Install WSL2

First, you need to install WSL2. Follow this guide to set up WSL2 on your Windows machine. We recommend using Ubuntu as your Linux distribution.

1. Update Packages

Open your WSL terminal (Ubuntu) and update your packages:

sh sudo apt update

1. Install NVM and Node.js

NVM (Node Version Manager) helps manage multiple versions of Node.js. Install NVM and Node.js 18 (the stable version):

```sh curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.2/install.s