github.com/dathere/qsv @21.1.0

qsv: Blazing-fast Data-Wrangling toolkit

	Table of Contents

Hi-ho "Quicksilver" away!

_{^{original logo details * Base AI-reimagined logo * Event logo archive}}

|qsv is a data-wrangling toolkit for querying, slicing,

sorting, analyzing, filtering, enriching, transforming,

validating, joining, formatting, converting, chatting,

FAIRifying & documenting tabular data (CSV, Excel, etc).

Commands are simple, composable & "blazing fast".

• Commands

• Installation: CLI • MCP Server • Cowork Plugin

• Whirlwind Tour / Notebooks / Lessons & Exercises

• FAQ

• Performance Tuning

• 👉 Benchmarks 🚀

• Environment Variables

• Feature Flags

• Goals/Non-goals

• Testing

• NYC SOD 2022/csv,conf,v8/PyConUS 2025/

   csv,conf,v9/NYC SOD 2026

• "Have we achieved ACI?" series - 1 • 2 • 3

• Sponsor

Try it out at qsv.dathere.com!

📇🧠🤖🚀🔣👆⛩️ | Apply series of string, date, math & currency transformations to given CSV column/s. It also has some basic NLP functions (similarity, sentiment analysis, profanity, eudex, language & name gender) detection. Its summarize subcommand condenses a column or group of columns using an OpenAI API-compatible LLM (local or commercial) with customizable, Mini Jinja-templated per-record prompts. | | applydp✨

📇🚀🔣👆 | applydp is a slimmed-down version of apply with only Datapusher+ relevant subcommands/operations (qsvdp binary variant only). | | behead | Drop headers from a CSV. | | blake3

🚀 | Compute or check BLAKE3 hashes of files. | | cat

🗄️ | Concatenate CSV files by row or by column. | | clipboard✨

🖥️ | Provide input from the clipboard or save output to the clipboard. | | color✨

🤯🐻‍❄️🖥️ | Outputs tabular data as a pretty, colorized table that always fits into the terminal. Apart from CSV and its dialects, Arrow, Avro/IPC, Parquet, JSON array & JSONL formats are supported with the "polars" feature. | | count

📇🐻‍❄️🏎️ | Count the rows and optionally compile record width statistics of a CSV file. (11.87 seconds for a 15gb, 28m row NYC 311 dataset without an index. Instantaneous with an index.) If the polars feature is enabled, uses Polars' multithreaded, mem-mapped CSV reader for fast counts even without an index | | datefmt

📇🚀👆 | Formats recognized date fields (19 formats recognized) to a specified date format using strftime date format specifiers. | | dedup

🤯🚀👆 | Remove duplicate rows (See also extdedup, extsort, sort & sortcheck commands). | | describegpt

📇🗃️🤖🌐🪄📚⛩️ | Infer a "neuro-symbolic" Data Dictionary, Description & Tags or ask questions about a CSV with a configurable, Mini Jinja prompt file, using any OpenAI API-compatible LLM, including local LLMs.

(e.g. Markdown, JSON, TOON, JSON Schema, Semantic Markdown, Everything, Spanish, Mandarin, Controlled Tags;

--prompt "What are the top 10 complaint types by community board & borough by year?" - deterministic, hallucination-free SQL RAG result; iterative, session-based SQL RAG refinement - refined SQL RAG result) | | diff

🚀 | Find the difference between two CSVs with ludicrous speed!

e.g. compare two CSVs with 1M rows x 9 columns in under 600ms! | | edit | Replace the value of a cell specified by its row and column. | | enum

👆 | Add a new column enumerating rows by adding a column of incremental or uuid identifiers. Can also be used to copy a column or fill a new column with a constant value. | | excel

🚀 | Exports a specified Excel/ODS sheet to a CSV file. | | exclude

📇👆 | Removes a set of CSV data from another set based on the specified columns. | | explode

🔣👆 | Explode rows into multiple ones by splitting a column value based on the given separator. The inverse of implode. | | extdedup

👆 | Remove duplicate rows from an arbitrarily large CSV/text file using a memory-mapped, on-disk hash table. Unlike the dedup command, this command does not load the entire file into memory nor does it sort the deduped file. | | extsort

📇🚀👆 | Sort an arbitrarily large CSV/text file using a multithreaded external merge sort algorithm. | | fetch✨

📇🧠🌐 | Send/Fetch data to/from web services for every row using HTTP Get. Comes with HTTP/2 adaptive flow control, jaq JSON query language support, dynamic throttling (RateLimit) & caching with available persistent caching using Redis or a disk-cache. | | fetchpost✨

📇🧠🌐⛩️ | Similar to fetch, but uses HTTP Post (HTTP GET vs POST methods). Supports HTML form (application/x-www-form-urlencoded), JSON (application/json) and custom content types - with the ability to render payloads using CSV data using the Mini Jinja template engine. | | fill

👆 | Fill empty values. | | fixlengths | Force a CSV to have same-length records by either padding or truncating them. | | flatten | A flattened view of CSV records. Useful for viewing one record at a time.

e.g. qsv slice -i 5 data.csv \| qsv flatten. | | fmt | Reformat a CSV with different delimiters, record terminators or quoting rules. (Supports ASCII delimited data.) | | foreach✨ | Execute a shell command once per record in a given CSV file. | | frequency

📇😣🏎️👆🪄 | Build frequency distribution tables of each column. Uses multithreading to go faster if an index is present (Examples: CSV JSON TOON). | | get✨

📇🧠🌐 | Get tabular data from local files, URLs (http/https & dathere://) & CKAN (ckan://) into a managed, queryable disk cache - with conditional revalidation (ETag/Last-Modified), transparent zstd compression, BLAKE3 hashing & automatic indexing. Cached resources are reusable by ANY qsv command via the dc: prefix (e.g. qsv stats dc:data.csv), with stale entries auto-refreshed. Efficiently seeds luau lookup tables, validate dynamicEnum reference data & speeds up Datapusher+ harvesting. | | geocode✨

📇🧠🚀🌐🔣👆🌎 | Geocodes a location against an updatable local copy of the Geonames cities & the Maxmind GeoLite2 databases — with caching and multi-threading, this offline path geocodes up to 360,000 records/sec! Can also geocode online (forward & reverse) via the OpenCage geocoder. | | geoconvert✨

🌎 | Convert between various spatial formats and CSV/SVG including GeoJSON, SHP, and more. | | headers

🗄️ | Show the headers of a CSV. Or show the intersection of all headers between many CSV files. | | implode

😣👆 | Implode rows by grouping on key column(s) and joining a value column with a given separator. The inverse of explode. | | index | Create an index (📇) for a CSV. This is very quick (even the 15gb, 28m row NYC 311 dataset takes all of 14 seconds to index) & provides constant time indexing/random access into the CSV. With an index, count, sample & slice work instantaneously; random access mode is enabled in luau; and multithreading (🏎️) is enabled for the frequency, split, stats & `sc