llm.codes 📖 - Transform Developer Documentation for AI Agents

A high-performance web service that converts JavaScript-heavy documentation sites into clean, LLM-optimized Markdown. Built specifically to solve the problem of AI agents being unable to parse modern documentation sites that rely heavily on client-side rendering.

Why llm.codes exists: Modern AI agents like Claude Code struggle with JavaScript-heavy documentation sites, particularly Apple's developer docs. This tool bridges that gap by converting dynamic content into clean, parseable Markdown that AI agents can actually use.

📖 Read the full story: How llm.codes Transforms Developer Documentation for AI Agents

Technical Architecture

Core Problem Solved

Modern documentation sites (especially Apple's) use heavy JavaScript rendering that makes content invisible to AI agents. llm.codes solves this by:

• Using Firecrawl's hosted browser, or opt-in self-hosted Playwright, to execute JavaScript and capture fully-rendered content
• Converting dynamic HTML to clean, semantic Markdown
• Removing noise (navigation, footers, duplicate content) that wastes AI context tokens
• Providing parallel URL processing for efficient multi-page documentation crawling

Key Features

• Parallel Processing: Fetches up to 10 URLs concurrently using bounded promises
• Smart Caching: Redis-backed 30-day cache reduces API calls and improves response times
• Content Filtering: Multiple filtering strategies to remove:
• Navigation elements and boilerplate
• Platform availability strings (iOS 14.0+, etc.)
• Duplicate content across pages
• Empty sections and formatting artifacts
• Recursive Crawling: Configurable depth-first crawling with intelligent link extraction
• Firecrawl Crawl Mode: Optional crawl API path with server-side depth/limit enforcement and SSE progress
• Code-Only Output: Extract fenced examples only and skip pages without code blocks
• Browser Notifications: Web Notifications API integration for background processing alerts
• URL State Management: Query parameter-based URL sharing for easy documentation links

Live Demo

🚀 Try it now at llm.codes

Experience the tool instantly without any setup required.

Quick Start

Prerequisites

• Node.js 24+
• pnpm 10+
• Firecrawl API key for the default provider, or self-hosted Chromium for the Playwright provider

Installation

1. Clone the repository:

git clone https://github.com/amantus-ai/llm-codes.git
cd llm-codes

1. Install dependencies:

pnpm install

1. Create a .env.local file:

cp .env.local.example .env.local

1. Add your scrape provider configuration to .env.local:

# Default hosted provider
SCRAPE_PROVIDER=firecrawl
FIRECRAWL_API_KEY=your_api_key_here

# Self-hosted alternative
# SCRAPE_PROVIDER=playwright
# Run once: pnpm exec playwright install chromium

# Optional - Redis Cache (Recommended for production)
UPSTASH_REDIS_REST_URL=https://your-redis-instance.upstash.io
UPSTASH_REDIS_REST_TOKEN=your_redis_token_here

# Optional - Cache Admin
CACHE_ADMIN_KEY=your_secure_admin_key_here

1. Run the development server:

pnpm run dev

1. Open http://localhost:3000

Deployment

Deploy to Vercel

The easiest way to deploy is using Vercel:

1. Click the button above
2. Create a new repository
3. Add your FIRECRAWL_API_KEY environment variable
4. Deploy!

Manual Deployment

1. Push to your GitHub repository
2. Import project on Vercel
3. Add environment variables:
4. SCRAPE_PROVIDER: firecrawl by default, or playwright for self-hosted Node/Docker deployments
5. FIRECRAWL_API_KEY: Your Firecrawl API key (required when SCRAPE_PROVIDER=firecrawl)
6. UPSTASH_REDIS_REST_URL: Your Upstash Redis URL (optional)
7. UPSTASH_REDIS_REST_TOKEN: Your Upstash Redis token (optional)
8. CACHE_ADMIN_KEY: Admin key for cache endpoints (optional)
9. Deploy

Usage

1. Enter URL: Paste any documentation URL
2. Most documentation sites are automatically supported through pattern matching

3. Click "Learn more" to see the supported URL patterns

4. Configure Options (click "Show Options"):

5. Crawl Depth: How deep to follow links (0 = main page only, max 5)
6. Max URLs: Maximum number of pages to process (1-2000, default 200)
7. Filter URLs: Remove hyperlinks from content (recommended for LLMs)
8. Deduplicate Content: Remove duplicate paragraphs to save tokens
9. Filter Availability: Remove platform availability strings (iOS 14.0+, etc.)
10. Extract Code Blocks Only: Include fenced examples only and skip pages without examples

11. Use Deep Crawl Mode: Use Firecrawl's crawl API with the selected depth and page limit. Turn this off when using SCRAPE_PROVIDER=playwright.

12. Process: Click "Process Documentation" and grant notification permissions if prompted

13. Monitor Progress:

14. Real-time progress bar shows completion percentage
15. Activity log displays detailed processing information

16. Browser notifications alert you when complete

17. Download: View statistics and download your clean Markdown file

Supported Documentation Sites

llm.codes uses intelligent pattern matching to support most documentation sites automatically. Rather than maintaining a list of thousands of individual sites, we use regex patterns to match common documentation URL structures.

Pattern-Based Matching

We support documentation sites that match these patterns:

1. Documentation Subdomains (docs.*, developer.*, learn.*, etc.)
2. Examples: docs.python.org, developer.apple.com, learn.microsoft.com

3. Pattern: Any subdomain like docs, developer, dev, learn, help, api, guide, wiki, or devcenter

4. Documentation Paths (/docs, /guide, /learn, etc.)

5. Examples: angular.io/docs, redis.io/docs, react.dev/learn

6. Pattern: URLs ending with paths like /docs, /documentation, /api-docs, /guides, /learn, /help, /stable, or /latest

7. Programming Language Sites (*js.org, *lang.org, etc.)

8. Examples: vuejs.org, kotlinlang.org, ruby-doc.org

9. Pattern: Domains ending with js, lang, py, or -doc followed by .org or .com

10. GitHub Pages (*.github.io)

11. Examples: Any GitHub Pages documentation site
12. Pattern: All subdomains of github.io

Explicit Exceptions

A small number of popular documentation sites don't follow standard patterns and are explicitly supported:

• Swift Package Index (swiftpackageindex.com)
• Flask (flask.palletsprojects.com)
• Material-UI (mui.com/material-ui)
• pip (pip.pypa.io/en/stable)
• PHP (www.php.net/docs.php)
• docs.rs (docs.rs)
• Zulip API (zulip.com/api)
• Nutrient API (www.nutrient.io/api)
• Playdate SDK (sdk.play.date)
• MusicKit JS (js-cdn.music.apple.com/musickit)
• React Spring (react-spring.io)
• Valtio (valtio.pmnd.rs)

Adding New Sites

Most documentation sites are automatically supported! If your site follows standard documentation URL patterns (like having /docs in the path or docs. as a subdomain), it should work without any changes.

If you find a documentation site that isn't supported, please open an issue and we'll either adjust our patterns or add it as an exception.

Configuration Options

API Architecture

POST /api/scrape

The core API endpoint that handles documentation conversion.

Request Flow:

1. URL validation against documentation URL patterns and explicit exceptions
2. Cache check (Redis/in-memory with 30-day TTL)
3. Provider call with optimized scraping parameters. Firecrawl is the default; SCRAPE_PROVIDER=playwright launches self-hosted Chromium.
4. Content post-processing and filtering
5. Response with markdown and cache status

Request Body:

{
  "url": "https://developer.apple.com/documentation/swiftui",
  "action": "scrape"
}

Response:

{
  "success": true,
  "data": {
    "markdown": "# SwiftUI Documentation\n\n..."
  },
  "cached": false
}

Error Handling:

• Domain validation errors (400)
• Provider/API errors (500)
• Network timeouts (504)
• Rate limiting (429)

Tech Stack

• Framework: Next.js 16 with App Router
• Language: TypeScript
• Styling: Tailwind CSS v4
• API: Firecrawl by default; opt-in self-hosted Playwright extraction for Node/Docker deployments
• Cache: Upstash Redis for distributed caching
• Deployment: Vercel
• Development: Turbopack for fast refreshes

Project Structure

llm-codes/
├── src/
│   ├── app/
│   │   ├── api/
│   │   │   ├── scrape/                # Single-page scrape endpoint
│   │   │   ├── crawl/                 # Firecrawl crawl endpoints
│   │   │   └── cache/                 # Cache stats endpoint
│   │   ├── globals.css                # Global styles & Tailwind
│   │   ├── layout.tsx                 # Root layout
│   │   ├── page.tsx                   # Main page component
│   │   └── icon.tsx                   # Dynamic favicon
│   ├── constants.ts                   # Configuration constants
│   ├── utils/                         # Utility functions
│   │   ├── code-extraction.ts         # Code-block extraction
│   │   ├── content-processing.ts      # Content cleaning logic
│   │   ├── file-utils.ts              # File handling
│   │   ├── result-processing.ts       # Final output shaping
│   │   ├── url-utils.ts               # URL validation & handling
│   │   └── __tests__/                 # Utility tests
│   ├── lib/                           # Scrape providers, cache, retry, and HTTP clients
│   └── test/
│       └── setup.ts                   # Test configuration
├── public/
│   ├── logo.png                       # App icon
│   └── manifest.json                  # PWA manifest
├── next.config.js                     # Next.js configuration
├── postcss.config.js                  # PostCSS with Tailwind v4
├── tsconfig.json                      # TypeScript configuration
├── vitest.config.ts                   # Vitest test configuration
├── spec.md                            # Detailed specification
└── package.json                       # Dependencies

Technical Implementation Details

Content Processing Pipeline

1. URL Extraction: Custom regex patterns extract links from markdown and HTML
2. Domain-Specific Filtering: Each documentation site has custom rules for link following
3. Parallel Batch Processing: URLs processed in batches of 10 for optimal performance
4. Content Deduplication: Hash-based paragraph and section deduplication
5. Multi-Stage Filtering: Sequential filters for URLs, navigation, boilerplate, and platform strings

Performance Optimizations

• Batched API Calls: Reduces provider latency by processing multiple URLs per request
• Progressive Loading: UI updates with real-time progress during long crawls
• Smart Link Extraction: Only follows relevant documentation links based on URL patterns
• Client-Side Caching: Browser-based result caching for repeat operations

Testing Strategy

# Run all tests once
pnpm run test:run

# Run tests with UI
pnpm run test:ui

# Run tests with coverage
pnpm run test:coverage

# Type checking
pnpm run type-check

# Full local gate
pnpm run verify

# Live Firecrawl mode smoke without a local browser
pnpm run verify:modes:live

Tests cover:

• URL validation and domain filtering
• Content processing and deduplication
• API error handling
• Cache behavior
• UI component interactions

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Troubleshooting

Notifications not working?

• Check browser permissions for notifications
• Ensure you're using a supported browser (Chrome, Firefox, Safari 10.14+, Edge)
• Try resetting notification permissions in browser settings

API Rate Limits?

The app includes a 30-day cache to minimize API calls. If you're hitting rate limits:

• Reduce crawl depth
• Lower maximum URLs
• Wait for cached results
• Consider setting up Redis cache for better performance

Redis Cache Setup

For production use, we recommend setting up Redis cache:

1. Sign up for [Ups