Marmot v2

What & Why?

Marmot v2 is a leaderless, distributed SQLite replication system built on a gossip-based protocol with distributed transactions and eventual consistency.

Key Features: - Leaderless Architecture: No single point of failure - any node can accept writes - MySQL Protocol Compatible: Connect with any MySQL client (DBeaver, MySQL Workbench, mysql CLI) - WordPress Compatible: Full MySQL function support for running distributed WordPress - Distributed Transactions: Percolator-style write intents with conflict detection - Multi-Database Support: Create and manage multiple databases per cluster - DDL Replication: Distributed schema changes with automatic idempotency and cluster-wide locking - Production-Ready SQL Parser: Powered by rqlite/sql AST parser for MySQL→SQLite transpilation - CDC-Based Replication: Row-level change data capture for consistent replication

Why Marmot?

The Problem with Traditional Replication

MySQL active-active requires careful setup of replication, conflict avoidance, and monitoring. Failover needs manual intervention. Split-brain scenarios demand operational expertise. This complexity doesn't scale to edge deployments.

Marmot's Approach

• Zero operational overhead: Automatic recovery from split-brain via eventual consistency + anti-entropy
• No leader election: Any node accepts writes, no failover coordination needed
• Direct SQLite access: Clients can read the local SQLite file directly for maximum performance
• Tunable consistency: Choose ONE/QUORUM/ALL per your latency vs durability needs

Why MySQL Protocol?

• Ecosystem compatibility - existing drivers, ORMs, GUI tools work out-of-box
• Battle-tested wire protocol implementations
• Run real applications like WordPress without modification

Ideal Use Cases

Marmot excels at read-heavy edge scenarios:

When to Consider Alternatives

• Strong serializability required → CockroachDB, Spanner
• Single-region high-throughput → PostgreSQL, MySQL directly
• Large datasets (>100GB) → Sharded solutions

Quick Start

# Start a single-node cluster
./marmot-v2

# Or run as daemon (background)
./marmot-v2 -daemon -pid-file=/tmp/marmot/marmot.pid

# Connect with MySQL client
mysql -h localhost -P 3306 -u root

# Or use DBeaver, MySQL Workbench, etc.

Testing Replication

# Test DDL and DML replication across a 2-node cluster
./scripts/test-ddl-replication.sh

# This script will:
# 1. Start a 2-node cluster
# 2. Create a table on node 1 and verify it replicates to node 2
# 3. Insert data on node 1 and verify it replicates to node 2
# 4. Update data on node 2 and verify it replicates to node 1
# 5. Delete data on node 1 and verify it replicates to node 2

# Manual cluster testing
./examples/start-seed.sh              # Start seed node (port 8081, mysql 3307)
./examples/join-cluster.sh 2 localhost:8081  # Join node 2 (port 8082, mysql 3308)
./examples/join-cluster.sh 3 localhost:8081  # Join node 3 (port 8083, mysql 3309)

# Connect to any node and run queries
mysql --protocol=TCP -h localhost -P 3307 -u root
mysql --protocol=TCP -h localhost -P 3308 -u root

# Cleanup
pkill -f marmot-v2

WordPress Support

Marmot can run distributed WordPress with full database replication across nodes. Each WordPress instance connects to its local Marmot node, and all database changes replicate automatically.

MySQL Compatibility for WordPress

Marmot implements MySQL functions required by WordPress:

Quick Start: 3-Node WordPress Cluster

cd examples/wordpress-cluster
./run.sh up

This starts: - 3 Marmot nodes with QUORUM write consistency - 3 WordPress instances each connected to its local Marmot node

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│ WordPress-1 │  │ WordPress-2 │  │ WordPress-3 │
│ :9101       │  │ :9102       │  │ :9103       │
└──────┬──────┘  └──────┬──────┘  └──────┬──────┘
       ▼                ▼                ▼
┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│  Marmot-1   │◄─┤  Marmot-2   │◄─┤  Marmot-3   │
│ MySQL: 9191 │  │ MySQL: 9192 │  │ MySQL: 9193 │
└─────────────┘  └─────────────┘  └─────────────┘
       └──────────────┴──────────────┘
              QUORUM Replication

Test it: 1. Open http://localhost:9101 - complete WordPress installation 2. Open http://localhost:9102 or http://localhost:9103 3. See your content replicated across all nodes!

Commands:

./run.sh status   # Check cluster health
./run.sh logs-m   # Marmot logs only
./run.sh logs-wp  # WordPress logs only
./run.sh down     # Stop cluster

Production Considerations for WordPress

• Media uploads: Use S3/NFS for shared media storage (files not replicated by Marmot)
• Sessions: Use Redis or database sessions for sticky-session-free load balancing
• Caching: Each node can use local object cache (Redis/Memcached per region)

Architecture

Marmot v2 uses a fundamentally different architecture from other SQLite replication solutions:

vs. rqlite/dqlite/LiteFS: - ❌ They require a primary node for all writes - ✅ Marmot allows writes on any node - ❌ They use leader election (Raft) - ✅ Marmot uses gossip protocol (no leader) - ❌ They require proxy layer or page-level interception - ✅ Marmot uses MySQL protocol for direct database access

How It Works: 1. Write Coordination: 2PC (Two-Phase Commit) with configurable consistency (ONE, QUORUM, ALL) 2. Conflict Resolution: Last-Write-Wins (LWW) with HLC timestamps 3. Cluster Membership: SWIM-style gossip with failure detection 4. Data Replication: Full database replication - all nodes receive all data 5. DDL Replication: Cluster-wide schema changes with automatic idempotency

Comparison with Alternatives

DDL Replication

Marmot v2 supports distributed DDL (Data Definition Language) replication without requiring master election:

How It Works

1. Cluster-Wide Locking: Each DDL operation acquires a distributed lock per database (default: 30-second lease)
2. Prevents concurrent schema changes on the same database
3. Locks automatically expire if a node crashes

4. Different databases can have concurrent DDL operations

5. Automatic Idempotency: DDL statements are automatically rewritten for safe replay ```sql CREATE TABLE users (id INT) → CREATE TABLE IF NOT EXISTS users (id INT)

DROP TABLE users → DROP TABLE IF EXISTS users ```

1. Schema Version Tracking: Each database maintains a schema version counter
2. Incremented on every DDL operation
3. Exchanged via gossip protocol for drift detection

4. Used by delta sync to validate transaction applicability

5. Quorum-Based Replication: DDL replicates like DML through the same 2PC mechanism

6. No special master node needed
7. Works with existing consistency levels (QUORUM, ALL, etc.)

Configuration

[ddl]
# DDL lock lease duration (seconds)
lock_lease_seconds = 30

# Automatically rewrite DDL for idempotency
enable_idempotent = true

Best Practices

• ✅ Do: Execute DDL from a single connection/node at a time
• ✅ Do: Use qualified table names (mydb.users instead of users)
• ⚠️ Caution: ALTER TABLE is less idempotent - avoid replaying failed ALTER operations
• ❌ Don't: Run concurrent DDL on the same database from multiple nodes

CDC-Based Replication

Marmot v2 uses Change Data Capture (CDC) for replication instead of SQL statement replay:

How It Works

1. Row-Level Capture: Instead of replicating SQL statements, Marmot captures the actual row data changes (INSERT/UPDATE/DELETE)
2. Binary Data Format: Row data is serialized as CDC messages with column values, ensuring consistent replication regardless of SQL dialect
3. Deterministic Application: Row data is applied directly to the target database, avoiding parsing ambiguities

Benefits

• Consistency: Same row data applied everywhere, no SQL parsing differences
• Performance: Binary format is more efficient than SQL text
• Reliability: No issues with SQL syntax variations between MySQL and SQLite

Row Key Extraction

For UPDATE and DELETE operations, Marmot automatically extracts row keys: - Uses PRIMARY KEY columns when available - Falls back to ROWID for tables without explicit primary key - Handles composite primary keys correctly

CDC Publisher

Marmot can publish CDC events to external messaging systems, enabling real-time data pipelines, analytics, and event-driven architectures. Events follow the Debezium specification for maximum compatibility with existing CDC tooling.

Features

• Debezium-Compatible Format: Events conform to the Debezium event structure, compatible with Kafka Connect, Flink, Spark, and other CDC consumers
• Multi-Sink Support: Publish to multiple destinations simultaneously (Kafka, NATS)
• Glob-Based Filtering: Filter which tables and databases to publish
• Automatic Retry: Exponential backoff with configurable limits
• Persistent Cursors: Survives restarts without losing position

Configuration

[publisher]
enabled = true

[[publisher.sinks]]
name = "kafka-main"
type = "kafka"                    # "kafka" or "nats"
format = "debezium"               # Debezium-compatible JSON format
brokers = ["localhost:9092"]      # Kafka broker addresses
topic_prefix = "marmot.cdc"       # Topics: {prefix}.{database}.{table}
filter_tables = ["*"]             # Glob patterns (e.g., "users", "order_*")
filter_databases = ["*"]          # Glob patterns (e.g., "prod_*")
batch_size = 100                  # Events per poll cycle
poll_interval_ms = 10             # Polling interval

# NATS sink example
[[publisher.sinks]]
name = "nats-events"
type = "nats"
format = "debezium"
nats_url = "nats://localhost:4222"
topic_prefix = "marmot.cdc"
filter_tables = ["*"]
filter_databases = ["*"]

Event Format

Events follow the Debezium envelope structure:

{
  "schema": { ... },
  "payload": {
    "before": null,
    "after": {"id": 1, "name": "alice", "email": "alice@example.com"},
    "source": {
      "version": "2.0.0",
      "connector": "marmot",
      "name": "marmot",
      "ts_ms": 1702500000000,
      "db": "myapp",
      "table": "users"
    },
    "op": "c",
    "ts_ms": 1702500000000
  }
}

Operation Types (per Debezium spec): | Operation | op | before | after | |-----------|------|----------|---------| | INSERT | c (create) | null | row data | | UPDATE | u (update) | old row | new row | | DELETE | d (delete) | old row | null |

Topic Naming

Topics follow the pattern: {topic_prefix}.{database}.{table}

Examples: - marmot.cdc.myapp.users - marmot.cdc.myapp.orders - marmot.cdc.analytics.events

Use Cases

• Real-Time Analytics: Stream changes to data warehouses (Snowflake, BigQuery, ClickHouse)
• Event-Driven Microservices: Trigger actions on data changes
• Cache Invalidation: Keep caches in sync with database changes
• Audit Logging: Capture all changes for compliance
• Search Indexing: Keep Elasticsearch/Algolia in sync

For more details, see the Integrations documentation.

Edge Deployment Patterns

Lambda Sidecar

Deploy Marmot as a lightweight regional replica alongside Lambda functions: - Local SQLite reads (sub-ms latency) - Writes replicate to cluster - No cold-start database connections

Read-Only Regional Replicas

Scale reads globally with replica mode and transparent failover:

```toml [replica] enable