TinyETL

Fast, zero-config ETL in a single binary

Transform and move data between any format or database instantly. No dependencies, no config files, just one command.

# MySQL → Parquet with inline transformation 
tinyetl "mysql://user:@host/db#orders" orders.parquet \
  --transform "total_usd=row.amount * row.exchange_rate"

# Stream 100k+ rows/sec from CSV → SQLite
tinyetl large_dataset.csv results.db --batch-size 50000

# Download & convert web data
tinyetl "https://api.data.gov/export.json" analysis.parquet

Why TinyETL?

✅ Single 15 MB binary — no dependencies, no installation headaches
✅ 180k+ rows/sec streaming — handles massive datasets efficiently
✅ Zero configuration — automatic schema detection and table creation (override with schema and config files in yaml)
Note: Auto-inferred schemas default all columns to nullable for safety

✅ Lua transformations — powerful data transformations
✅ Universal connectivity — CSV, JSON, Parquet, Avro, MySQL, PostgreSQL, SQLite, DuckDB, MSSQL, ODBC. Coming soon: Snowflake, Databricks, OneLake

✅ Cross-platform — Linux, macOS, Windows ready

Quick Install

Download the binary (recommended):

Visit the releases page and download the appropriate binary for your platform: - Linux x64, Linux ARM64 - macOS Intel, macOS Apple Silicon
- Windows x64, Windows ARM64

Or install with Cargo (builds from source):

cargo install tinyetl

Verify installation:

tinyetl --version

Get Started in 30 Seconds

# File format conversion (auto-detects schemas)
tinyetl data.csv output.parquet
tinyetl data.json analysis.db

# Database to database 
tinyetl "postgresql://user:@host/db#users" "mysql://user:@host/db#users"

# Transform while transferring
tinyetl sales.csv results.db --transform "profit=row.revenue - row.costs; margin=profit/revenue"

# Process large datasets efficiently  
tinyetl huge_dataset.csv output.parquet --batch-size 100000

# Download and convert web data
tinyetl "https://example.com/api/export" local_data.json --source-type=csv

# Run complex ETL jobs from configuration files
tinyetl run my_etl_job.yaml

Usage

Usage: tinyetl [OPTIONS] 

 <TARGET>

Direct Transfer:


  Source connection string (file path or connection string)
  <TARGET>  Target connection string (file path or connection string)

Options:
      --infer-schema             Auto-detect columns and types
      --schema-file <FILE>       Path to schema file (YAML) to override auto-detection
      --batch-size <BATCH_SIZE>  Number of rows per batch [default: 10000]
      --preview <N>              Show first N rows and inferred schema without copying
      --dry-run                  Validate source/target without transferring data
      --log-level <LOG_LEVEL>    Log level: info, warn, error [default: info]
      --skip-existing            Skip rows already in target if primary key detected
      --truncate                 Truncate target before writing (overrides append-first behavior)
      --transform-file <FILE>    Path to Lua file containing a 'transform' function
      --transform <EXPRESSIONS>  Inline transformation expressions (semicolon-separated, e.g., "new_col=row.old_col * 2; name=row.first .. ' ' .. row.last")
      --source-type <TYPE>       Force source file type (csv, json, parquet) - useful for HTTP URLs without clear extensions
      --source-secret-id <ID>    Secret ID for source password (resolves to TINYETL_SECRET_{id})
      --dest-secret-id <ID>      Secret ID for destination password (resolves to TINYETL_SECRET_{id})
  -h, --help                     Print help
  -V, --version                  Print version

Config File Mode:
  run <CONFIG_FILE>  Run ETL job from YAML configuration file

Config File Generation Modes:
  generate-config [OPTIONS] 

 <TARGET>  Generate a YAML configuration file from CLI arguments and output to STDOUT
  generate-default-config                      Generate a default YAML configuration example and output to STDOUT

Basic usage examples:

# Local file operations
tinyetl data.csv output.db
tinyetl data.csv output.parquet
tinyetl data.csv output.avro
tinyetl data.json output.csv
tinyetl data.avro output.json

# Download from web
tinyetl "https://example.com/data.csv" output.json
tinyetl "https://api.example.com/export" data.csv --source-type=csv

# Secure file transfer via SSH
tinyetl "ssh://user@server.com/data.csv" output.parquet

# Database operations
tinyetl data.csv "postgresql://user:pass@localhost/mydb#customers"
tinyetl data.csv "mysql://user:pass@localhost:3306/mydb#customers"
tinyetl "sqlite:///source.db#users" output.csv

# Data inspection and validation
tinyetl data.csv output.db --preview 10
tinyetl data.csv output.db --dry-run

# Advanced options
tinyetl data.csv output.db --batch-size 5000
tinyetl data.csv output.db --preview 10

# Transfer with custom batch size
tinyetl data.csv output.db --batch-size 5000

# Dry run to validate without transferring
tinyetl data.csv output.db --dry-run

# Apply inline transformations
tinyetl data.csv output.db --transform "full_name=row.first_name .. ' ' .. row.last_name; age_next_year=row.age + 1"

# Apply transformations from Lua file
tinyetl data.csv output.db --transform-file transform.lua

Supported Data Sources and Targets

TinyETL supports two main categories of data sources and targets:

File-Based Sources (Multiple Protocols)

File Formats: - CSV - Comma-separated values - JSON - JavaScript Object Notation (array of objects) - Parquet - Columnar storage format - Avro - Binary serialization format with schema evolution

Access Protocols: - Local Files - Direct file system access bash tinyetl data.csv output.json tinyetl data.csv output.avro tinyetl /path/to/file.parquet data.csv tinyetl data.avro output.json - HTTP/HTTPS - Download from web servers (supports authentication and custom headers via YAML config) bash tinyetl "https://example.com/data.csv" output.parquet tinyetl "https://api.example.com/export" data.csv --source-type=csv # For authenticated APIs, use YAML config (see HTTP Source Options section) - SSH/SCP - Secure file transfer bash tinyetl "ssh://user@server.com/data/file.csv" output.json tinyetl "ssh://user@server.com:2222/remote/data.parquet" local.csv

Protocol Features: - file:// - Local file system (default for simple paths) - http:// and https:// - Web downloads with progress tracking - Supports Basic and Bearer token authentication - Custom HTTP headers via YAML configuration - Environment variable substitution for secure credential management - ssh:// - Secure shell file transfer using SCP - --source-type parameter for format override (useful for URLs without clear extensions)

Database Sources

Supported Databases: - SQLite - Embedded database - PostgreSQL - Advanced open-source database - MySQL - Popular relational database - DuckDB - Embedded analytical database optimized for OLAP workloads - ODBC - Universal database connectivity (SQL Server, Oracle, DB2, and more)

Connection Examples:

# SQLite
tinyetl "sqlite:///path/to/db.sqlite#table" output.csv
tinyetl data.csv "sqlite:///output.db#customers"

# PostgreSQL  
tinyetl "postgresql://user:@localhost/mydb#orders" output.parquet
tinyetl data.csv "postgresql://user:@localhost/mydb#customers"

# MySQL
tinyetl "mysql://user:@localhost:3306/mydb#products" output.json
tinyetl data.csv "mysql://user:@localhost:3306/mydb#sales"

# DuckDB
tinyetl "products.duckdb#inventory" output.csv
tinyetl data.csv "analytics.duckdb#sales"

# ODBC - SQL Server (Windows with Trusted Auth)
tinyetl data.csv "odbc://Driver={ODBC Driver 18 for SQL Server};Server=localhost\MSSQLSERVER01;Database=master;Trusted_Connection=Yes;TrustServerCertificate=Yes#customers"

# ODBC - SQL Server (with username/password)
tinyetl data.csv "odbc://Driver={SQL Server};Server=localhost;Database=mydb;UID=sa;PWD=MyPass123#orders"

# ODBC - Reading from SQL Server
tinyetl "odbc://Driver={ODBC Driver 18 for SQL Server};Server=localhost;Database=master;Trusted_Connection=Yes#employees" output.csv

# DuckDB
tinyetl "products.duckdb#inventory" output.csv
tinyetl data.csv "analytics.duckdb#sales"

Source Type Override

When using HTTP/HTTPS or SSH protocols, URLs may not always indicate the file format clearly (e.g., API endpoints, URLs with query parameters). Use the --source-type parameter to explicitly specify the format:

# API endpoint that returns CSV data
tinyetl "https://api.example.com/export?format=csv&limit=1000" output.json --source-type=csv

# Google Drive download (no file extension in URL)
tinyetl "https://drive.google.com/uc?id=FILE_ID&export=download" data.csv --source-type=csv

# SSH file without clear extension
tinyetl "ssh://user@server.com/data/export_20241107" output.parquet --source-type=json

# Local files usually don't need source-type (auto-detected from extension)
tinyetl data.csv output.json  # No --source-type needed

Supported source types: csv, json, parquet, avro

Database Connection Strings

TinyETL uses standard database connection URLs with an optional table specification using the # separator.

PostgreSQL:

# Basic format
postgresql://username:password@hostname:port/database#table_name

# Examples
tinyetl data.csv "postgresql://user:@localhost/mydb#customers"
tinyetl data.csv "postgresql://admin:@db.example.com:5432/analytics#sales_data"

MySQL:

# Basic format  
mysql://username:password@hostname:port/database#table_name

# Examples
tinyetl data.csv "mysql://user:@localhost:3306/mydb#customers"
tinyetl data.csv "mysql://admin:@db.example.com:3306/analytics#sales_data"

# Default table name is 'data' if not specified
tinyetl data.csv "mysql://user:@localhost:3306/mydb"  # Creates table named 'data'

SQLite:

# File path (table name inferred from filename without extension)
tinyetl data.csv output.db              # Creates table named 'output'
tinyetl data.csv /path/to/database.db   # Creates table named 'database'

# Explicit table name using connection string format
tinyetl data.csv "sqlite:///path/to/database.db#custom_table"

DuckDB:

# File path (table name inferred from filename without extension)
tinyetl data.csv output.duckdb              # Creates table named 'output'
tinyetl data.csv /path/to/analytics.duckdb   # Creates table named 'analytics'

# Explicit table name using # separator
tinyetl data.csv "analytics.duckdb#sales_data"

# Reading from DuckDB table
tinyetl "products.duckdb#inventory" output.csv
tinyetl "analytics.duckdb#daily_sales" report.parquet

ODBC (Universal Database Connectivity):

ODBC provides connectivity to SQL Server, Oracle, DB2, and many other databases. The connection string format varies by driver.

```bash

SQL Server with Windows Authentication (Trusted Connection)

tinyetl data.csv "odbc://Driver={ODBC Driver 18 for SQL Server};Server=localhost\MSSQLSERVER01;Database=master;Trusted_Connection=Yes;TrustServerCertificate=Yes#customers"

SQL Server with SQL Authentication

tinyetl data.csv "odbc://Driver={SQL Server};Server=localhost;Database=mydb;UID=sa;PWD=MyPass123#orders"

SQL Server - Reading data

tinyetl "odbc://Driver={ODBC Driver 18 for SQL Server};Server=localhost;Database=mydb;Trusted_Connection=Yes#sales" output.csv

Oracle Database

tinyetl data.csv "odbc://Driver={Oracle ODBC Driver};Server=localhost:1521;Database=orcl;UID=user;PWD=pass#employees"

IBM DB2

tinyetl data.csv "odbc://Driver={IBM DB2 ODBC DRIVER};Database=sample;