Ducktape 🦆

Lightweight REST API for DuckDB with HTTP/2 streaming support.

What is ducktape?

Ducktape is a standalone microservice to:

• Append: Append rows directly into DuckDB by streaming NDJSON over HTTP/2.
• Query: Fetch rows from DuckDB.
• Execute: Run statements within a transaction.

Why? DuckDB's Go driver requires CGO, which breaks cross-compilation, complicates CI/CD, and bloats Docker images. Instead of rewriting the build pipelines for Transfer, we isolated DuckDB behind a network boundary.

The performance penalty is small—~90% of native throughput over the network. Pure Go apps stay portable; ducktape handles the CGO.

A native Go client library is included.

How it works

your service → stream NDJSON over HTTP/2 → ducktape → DuckDB (local file or MotherDuck)

Ducktape uses:

• HTTP/2 streaming for high-throughput ingestion
• DuckDB's Appender API for fast, type-aware row insertion
• NDJSON as a simple, language-agnostic wire format

If your app can produce NDJSON, it can talk to ducktape.

Performance

Benchmark	Throughput
In-process DuckDB append	~848 MiB/sec
Ducktape over HTTP/2	~757 MiB/sec

That's ~90% of native performance, even across the network. For real-time ingestion workloads, this was fast enough that we didn't need to embed DuckDB at all.

See BENCHMARKS.md for detailed results.

Quick start

Docker

docker pull artielabs/ducktape:latest
docker run -e DUCKTAPE_LOG="debug" --rm --publish 8080:8080 --volume $PWD:/data artielabs/ducktape:latest

# absolute path in DSN is required when ducktape runs in Docker and writing to local file
curl -X POST 'http://localhost:8080/api/query' \
--header 'X-DuckDB-Connection-String: /data/test.db' \
--header 'Content-Type: application/json' \
--data '{
    "Query": "CREATE TABLE test_file (id BIGINT);"
}'
# test.db will be created in your current working directory

Development

make start
# Or with debug logging
make debug
# Or manually
PORT=8080 DUCKTAPE_LOG=debug go run cmd/main.go

# Health check
curl http://localhost:8080/health

# Readiness check
curl http://localhost:8080/ready

Server runs on port 8080 by default.

API usage

Execute

Execute one or more SQL statements in a transaction:

curl -X POST http://localhost:8080/api/execute \
  -H "X-DuckDB-Connection-String: duck.db" \
  -H "Content-Type: application/json" \
  -d '{"statements": [
    {"query": "CREATE TABLE users (name TEXT, age INTEGER)"},
    {"query": "INSERT INTO users VALUES (?, ?)", "args": ["Alice", 30]},
    {"query": "INSERT INTO users VALUES (?, ?)", "args": ["Bob", 25]}
  ]}'

Query

curl -X POST http://localhost:8080/api/query \
  -H "X-DuckDB-Connection-String: duck.db" \
  -H "Content-Type: application/json" \
  -d '{"query": "SELECT * FROM users WHERE name = ?", "args": ["Alice"]}'

Query after running an init script

• This example uses an init script stored in the container to set up a connection to a self-hosted ducklake and then query it.

/sensitive/.duckdbrc

--put your own info in where there are <>

INSTALL httpfs;
LOAD httpfs;   

CREATE OR REPLACE SECRET ducklake_storage (
    TYPE s3,
    PROVIDER config,
    KEY_ID '<KEY_ID>,
    SECRET '<KEY_SECRET>',
    REGION 'us-east-1',
    ENDPOINT '<ENDPOINT>',
    URL_STYLE 'path',
    USE_SSL 'true'
);

INSTALL ducklake;
INSTALL postgres;

ATTACH 'ducklake:postgres:dbname=reporting_ducklake host=<HOST> port=<PORT> user=reporting_ducklake password=<PW>' 
AS reporting__prod
( DATA_PATH 's3://revo-reporting/ducklake/reporting/prod', METADATA_SCHEMA 'reporting__prod', OVERRIDE_DATA_PATH true ) 
;

USE reporting__prod;

• Call with connection string set as rcfile:<PATH TO FILE>

# this will run the given file to set up the ducklake connection prior to running 'show schemas;'
curl -sS -X POST 'http://localhost:8080/api/query' \
--header 'X-DuckDB-Connection-String: rcfile:/sensitive/.duckdbrc' \
--header 'Content-Type: application/json' \
--data '{
    "Query": "show schemas;"
}'

Append

Streams NDJSON data over HTTP/2. Each line is a RowMessage with a rv (row values) array. Use the Go client for streaming large datasets.

Configuration

• PORT: Server port (default: 8080)
• DUCKTAPE_LOG: Log level (debug, info, warn, error)
• DUCKTAPE_FLUSH_BYTES: Max payload size the Appender buffers before flushing, in bytes (default: 33554432, i.e. 32 MiB). Larger values amortise round-trip cost to remote destinations (e.g. MotherDuck) at the price of more in-process memory per stream.
• DUCKTAPE_FLUSH_ROWS: Max rows the Appender buffers before flushing (default: 100000).
• DUCKTAPE_SCANNER_BUFFER: Max size of a single NDJSON line read by bufio.Scanner, in bytes (default: 4194304, i.e. 4 MiB).

Go client

• Install Go module for client. bash go get github.com/artie-labs/ducktape/api
• Usage:

```go import "github.com/artie-labs/ducktape/api/pkg/ducktape"

client := ducktape.NewClient("http://localhost:8080") ```

• Client source code

License

MIT License. See LICENSE for details.