Spooled Backend

Job queue + worker coordination service built with Rust

Live Demo (SpriteForge) • Documentation • Website

Spooled is a high-performance, multi-tenant job queue system designed for reliability, observability, and horizontal scalability.

✨ Features

• High Performance: Built on Rust + Tokio + PostgreSQL with Redis caching (~28x faster auth)
• Optimized gRPC: HTTP/2 keepalive, TCP optimizations, and connection pooling for ~3x faster throughput
• Multi-Tenant: PostgreSQL Row-Level Security (RLS) for data isolation
• Observable: Prometheus metrics, Grafana dashboards, optional OpenTelemetry export (--features otel)
• Reliable: At-least-once processing with leases + retries (use idempotency keys for exactly-once effects)
• Real-Time: WebSocket + SSE for live job/queue updates
• Secure: Bcrypt API keys with Redis caching, JWT auth, HMAC webhook verification
• Scalable: Stateless API nodes (Kubernetes-friendly) + DB-backed locking (FOR UPDATE SKIP LOCKED)
• Scheduling: Cron-based recurring jobs with timezone support
• Workflows: Job dependencies with DAG execution
• Dual Protocol: REST API (:8080) + real gRPC (GRPC_PORT, default :50051) with streaming support
• Tier-Based Limits: Automatic enforcement across all endpoints (HTTP, gRPC, workflows, schedules)
• Dead Letter Queue: Automatic retry and purge operations for failed jobs
• Webhooks: Outgoing webhook delivery with automatic retries and status tracking
• Billing: Stripe integration for subscriptions and usage tracking

🐳 Quick Start with Docker

Pull and Run

# Pull the multi-arch image (supports amd64 and arm64)
docker pull ghcr.io/spooled-cloud/spooled-backend:latest

# Run with Docker Compose
curl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/docker-compose.prod.yml
curl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/.env.example
cp .env.example .env

# Generate secure secrets
export JWT_SECRET=$(openssl rand -base64 32)
export POSTGRES_PASSWORD=$(openssl rand -base64 16)
sed -i "s/your-jwt-secret-minimum-32-characters-long/$JWT_SECRET/" .env
sed -i "s/your_secure_password/$POSTGRES_PASSWORD/g" .env

# Start services
docker compose -f docker-compose.prod.yml up -d

# Verify
curl http://localhost:8080/health

Environment Variables

Variable	Required	Default	Description
DATABASE_URL	✅	-	PostgreSQL connection string
JWT_SECRET	✅	-	32+ char secret for JWT signing
ADMIN_API_KEY	❌	-	Key for admin portal access
REDIS_URL	❌	redis://localhost:6379	Redis for pub/sub & caching
RUST_ENV	❌	development	development/staging/production
REGISTRATION_MODE	❌	open	open/closed - controls public registration
PORT	❌	8080	REST API server port
GRPC_PORT	❌	50051	gRPC API server port
GRPC_TLS_ENABLED	❌	true (prod)	Enable TLS for gRPC (required for Cloudflare Tunnel)
GRPC_TLS_CERT_PATH	❌	/certs/grpc-cert.pem	Path to TLS certificate (PEM)
GRPC_TLS_KEY_PATH	❌	/certs/grpc-key.pem	Path to TLS private key (PEM)
METRICS_PORT	❌	9090	Prometheus metrics port
METRICS_TOKEN	❌	-	If set, requires Authorization: Bearer <token> for /metrics

Plan Limits via Environment Variables (Self-Hosted)

Spooled ships with sensible built-in plan defaults (Free/Starter/Pro/Enterprise), but you can override every plan limit via env vars.

Limits resolution order (lowest → highest precedence):

• Built-in defaults
• SPOOLED_PLAN_LIMITS_JSON (global per-tier JSON map)
• SPOOLED_PLAN_<TIER>_LIMITS_JSON (tier-specific JSON)
• SPOOLED_PLAN_<TIER>_<FIELD> (tier-specific individual fields)
• Organization custom_limits (DB, per-org override)

JSON overrides

• SPOOLED_PLAN_LIMITS_JSON: JSON object mapping tier → overrides (same keys as organizations.custom_limits)
• SPOOLED_PLAN_FREE_LIMITS_JSON, SPOOLED_PLAN_STARTER_LIMITS_JSON, SPOOLED_PLAN_PRO_LIMITS_JSON, SPOOLED_PLAN_ENTERPRISE_LIMITS_JSON

Example:

{
  "free": { "max_jobs_per_day": 5000, "max_payload_size_bytes": 131072 },
  "starter": { "max_active_jobs": 2000 },
  "enterprise": { "max_jobs_per_day": null }
}

Notes: - For optional limits (like max_jobs_per_day), null means unlimited.

Per-field overrides

You can override individual fields per tier with env vars:

• Limits (support unlimited / none / null / -1):
• SPOOLED_PLAN_<TIER>_MAX_JOBS_PER_DAY
• SPOOLED_PLAN_<TIER>_MAX_ACTIVE_JOBS
• SPOOLED_PLAN_<TIER>_MAX_QUEUES
• SPOOLED_PLAN_<TIER>_MAX_WORKERS
• SPOOLED_PLAN_<TIER>_MAX_API_KEYS
• SPOOLED_PLAN_<TIER>_MAX_SCHEDULES
• SPOOLED_PLAN_<TIER>_MAX_WORKFLOWS
• SPOOLED_PLAN_<TIER>_MAX_WEBHOOKS
• Sizes / rates / retention:
• SPOOLED_PLAN_<TIER>_MAX_PAYLOAD_SIZE_BYTES
• SPOOLED_PLAN_<TIER>_RATE_LIMIT_RPS
• SPOOLED_PLAN_<TIER>_RATE_LIMIT_BURST
• SPOOLED_PLAN_<TIER>_JOB_RETENTION_DAYS
• SPOOLED_PLAN_<TIER>_HISTORY_RETENTION_DAYS

Where <TIER> is one of: FREE, STARTER, PRO, ENTERPRISE.

🔧 Local Development

Prerequisites

• Rust 1.85+
• Docker & Docker Compose
• PostgreSQL 16+ (or use Docker)
• Redis 7+ (optional, for pub/sub)

Setup

# Clone repository
git clone https://github.com/spooled-cloud/spooled-backend.git
cd spooled-backend

# Start dependencies
docker compose up -d postgres redis

# Run migrations and start server
cargo run

# Run tests
cargo test

API Endpoints

Core Job Management

Method	Endpoint	Description
GET	/health	Health check
POST	/api/v1/jobs	Create a job (enforces plan limits)
GET	/api/v1/jobs	List jobs
POST	/api/v1/jobs/bulk	Bulk enqueue jobs (enforces plan limits)
POST	/api/v1/jobs/claim	Claim (lease) jobs for worker processing
POST	/api/v1/jobs/{id}/complete	Mark a job completed (worker ack)
POST	/api/v1/jobs/{id}/fail	Mark a job failed (worker nack)
POST	/api/v1/jobs/{id}/heartbeat	Extend a job lease (long-running jobs)
GET	/api/v1/jobs/stats	Get job statistics

Dead Letter Queue (DLQ)

Method	Endpoint	Description
GET	/api/v1/jobs/dlq	List jobs in dead letter queue
POST	/api/v1/jobs/dlq/retry	Retry jobs from DLQ (enforces plan limits)
POST	/api/v1/jobs/dlq/purge	Purge jobs from DLQ

Organizations

Method	Endpoint	Description
POST	/api/v1/organizations	Create organization (returns initial API key)
GET	/api/v1/organizations/usage	Get plan usage and limits
GET	/api/v1/organizations/check-slug	Check if slug is available
POST	/api/v1/organizations/generate-slug	Generate unique slug from name

Schedules & Workflows

Method	Endpoint	Description
POST	/api/v1/schedules	Create cron schedule
POST	/api/v1/schedules/{id}/trigger	Manually trigger schedule (enforces plan limits)
POST	/api/v1/workflows	Create workflow/DAG (enforces plan limits)

Webhooks

Method	Endpoint	Description
POST	/api/v1/outgoing-webhooks	Configure outgoing notifications
GET	/api/v1/outgoing-webhooks/{id}/deliveries	Get delivery history
POST	/api/v1/outgoing-webhooks/{id}/retry/{delivery_id}	Retry failed delivery
POST	/api/v1/webhooks/{org_id}/custom	Incoming webhook (ingestion → creates jobs)

Real-Time

Method	Endpoint	Description
GET	/api/v1/ws	WebSocket for real-time
GET	/api/v1/events	SSE stream of all events
GET	/api/v1/events/queues/{name}	SSE stream of queue updates
GET	/api/v1/events/jobs/{id}	SSE stream of job updates

Authentication

Method	Endpoint	Description
POST	/api/v1/auth/login	Exchange API key for JWT
POST	/api/v1/auth/refresh	Refresh JWT token
POST	/api/v1/auth/email/start	Start email-based login
POST	/api/v1/auth/email/verify	Verify email login code

Billing

Method	Endpoint	Description
GET	/api/v1/billing/status	Get billing status
POST	/api/v1/billing/portal	Create Stripe customer portal session

Admin API (requires X-Admin-Key header)

Method	Endpoint	Description
GET	/api/v1/admin/organizations	List all organizations
POST	/api/v1/admin/organizations	Create organization with plan tier
GET	/api/v1/admin/organizations/{id}	Get organization details
PATCH	/api/v1/admin/organizations/{id}	Update organization (plan, status)
DELETE	/api/v1/admin/organizations/{id}	Delete organization (soft or hard)
POST	/api/v1/admin/organizations/{id}/api-keys	Create API key for organization
POST	/api/v1/admin/organizations/{id}/reset-usage	Reset daily usage counters
GET	/api/v1/admin/stats	Platform-wide statistics
GET	/api/v1/admin/plans	List available plans with limits

Quick Examples

Create Your First Job

# 1. Create an organization (returns initial API key - save it!)
RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/organizations \
  -H "Content-Type: application/json" \
  -d '{"name": "My Company", "slug": "my-company"}')
echo "$RESPONSE"
# Save the api_key from the response - it's only shown once!
API_KEY=$(echo "$RESPONSE" | jq -r '.api_key')

# 2. Create a job using the API key
curl -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "queue_name": "emails",
    "payload": {"to": "user@example.com", "subject": "Hello!"},
    "priority": 0,
    "max_retries": 3
  }'

Create a Cron Schedule (Recurring Jobs)

# Run daily sales report every day at 9 AM
curl -X POST http://localhost:8080/api/v1/schedules \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "daily-sales-report",
    "cron_expression": "0 0 9 * * *",
    "timezone": "America/New_York",
    "queue_name": "reports",
    "payload_template": {"report_type": "daily_sales"}
  }'

Create a Workflow (Job Dependencies)

# User onboarding: create account → send email → setup defaults
curl -X POST http://localhost:8080/api/v1/workflows \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "user-onboarding",
    "jobs": [
      {
        "name": "create-account",
        "queue_name": "users",
        "payload": {"email": "user@example.com"}
      },
      {
        "name": "send-welcome",
        "queue_name": "emails",
        "depends_on": ["create-account"],
        "payload": {"template": "welcome"}
      },
      {
        "name": "setup-defaults",
        "queue_name": "users",
        "depends_on": ["create-account"],
        "payload": {"settings": {}}
      }
    ]
  }'

Configure Outgoing Webhooks (Notifications)

# Get notified in Slack when jobs fail
curl -X POST http://localhost:8080/api/v1/outgoing-webhooks \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Slack Alerts",
    "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
    "events": ["job.failed", "queue.paused"],
    "secret": "your-hmac-secret"
  }'

🔌 gRPC API

Spooled provides a real gRPC API using HTTP/2 + Protobuf for high-performance worker communication.

Endpoints

• Spooled Cloud (TLS): grpc.spooled.cloud:443
• Self-hosted / local: localhost:50051 (or whatever GRPC_PORT is set to)

gRPC TLS (Cloudflare Tunnel)

When using Cloudflare Tunnel with HTTPS origin, gRPC TLS is required because HTTP/2 needs TLS at the origin.

The production docker-compose includes: - TLS enabled by default (GRPC_TLS_ENABLED=true) - Self-signed certificates in ./certs/ (10-year validity) - Performance Optimized: HTTP/2 keepalives, TCP_NODELAY, and tuned connection windows

Cloudflare Tunnel Configuration: - Service Type: HTTPS - URL: backend:50051 - HTTP2 Connection: ON - No TLS Verify: ON (required for self-signed certs)

Note: Cloudflare Tunnel requires HTTPS for HTTP/2 (gRPC). You cannot use plaintext HTTP with gRPC through Cloudflare.

To disable TLS for local development (without Cloudflare):

GRPC_TLS_ENABLED=false cargo run

Proto Definition

The service definitions are in proto/spooled.proto:

```protobuf service QueueService { rpc Enqueue(EnqueueRequest) returns (EnqueueResponse); rpc Dequeue(DequeueRequest) returns (DequeueResponse); rpc Complete(CompleteRequest) returns (CompleteResponse); rpc Fail(FailRequest) returns (FailResponse); rpc RenewLease(RenewLeaseRequest) returns (RenewLeaseResponse); rpc GetJob(GetJobRequest) returns (GetJobResponse); rpc GetQueueStats(GetQueueStatsRequest) returns (GetQueueStatsResponse);

// Server-side streamin