parquet-tsdb-poc

POC for a parquet based TSDB in object storage.

Why

This project was inspired by this excellent talk by Shopify's Filip Petkovski. It is an attempt to build a service that can convert Prometheus TSDB blocks into parquet files and serve PromQL queries to backfill a Thanos deployment.

Developing

We recommend to use nix to fulfill all development dependencies. Visit Nix Download to get started. To activate the development environment simply run nix-shell in the project root.

• to build the binary run nix-shell --run 'make build'
• to run tests run nix-shell --run 'make test'

Running

Server

Once built, you can run the server using something like:

parquet-gateway serve \
    --storage.prefix my-prefix \
    --http.internal.port=6060 \
    --http.prometheus.port=9090 \
    --http.thanos.port=9091 \
    --block.syncer.interval=30m \
    --block.syncer.concurrency=32 \
    --block.discovery.interval=30m \
    --block.discovery.concurrency=32 \
    --query.external-label=prometheus=my-prometheus \
    --query.external-label=replica=ha-1

This will:

• load blocks from the .data/my-prefix directory
• expose internal metrics and readiness handlers on port 6060
• expose a subset of the Prometheus HTTP API on port 9090
• expose an Thanos Info and Query gRPC service on port 9091

You can now query it by pointing a Thanos Querier at it or through curl:

curl 'http://0.0.0.0:9000/api/v1/query' \
  -sq \
  -H 'content-type: application/x-www-form-urlencoded' \
  --data-urlencode 'query=vector(1)' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {},
        "value": [
          1741267893.103,
          "1"
        ]
      }
    ]
  }
}

Converter

To convert TSDB blocks in the .data/source directory that overlap 09/2021 and write the resulting parquet files into the .data/destination directory.

parquet-gateway convert \
    --tsdb.storage.prefix source \
    --parquet.storage.prefix destination \
    --convert.start=2021-09-01T00:00:00Z \
    --convert.end=2021-10-01T00:00:00Z