github.com/catenarytransit/catenary-backend @main

Architecture

Catenary Backend is a distributed system comprised of microservices operating in Kubernetes. The system is designed for fault tolerance, high-avaliability, and native execution speed in x86-64 using the Rust systems programming language.

The code is heavily commented, go to each folder in src for more information.

In production

• Maple: GTFS Downloader and ingestion engine into postgres
• Alpenrose: Distributed system to ingest GTFS-rt and other realtime data (Rose des Alpes); successor to Kactus.
• Aspen: Processing of realtime data and dynamic insertion into other engines. Submodule Pando is used for distribution management
• Aspen Leader: Coordinator/leader process for Aspen workers using etcd.
• Spruce: Websocket server for frontend to stream data to and from backend, including realtime locations, stop times
• Birch: HTTP API server
• OSM Station Import: Imports railway stations from OpenStreetMap PBF files for GTFS stop association

In active development

Loom (Line ordering optimisation maps)

• Globeflower: Equivalent to chapter 3: Line Graph Construction, by Patrick Brosi. (the C++ equivalent is topo)
• Harebell: Map tile geometry generator creating line ordering optimised graph maps (LOOM) MVT files. Equivalent to Chapter 4 & 6 combined, skipping 5. (the C++ equivalent is loom)

Routing

• Avens: OSM Preprocessor and graph generator for routing
• Gentian: Transit graph generation runner
• Edelweiss: Routing execution engine (in progress, see also our routing testbed

Citations

This project is built off the work of researchers in Germany and many other places, vielen dank!

• Round-Based Public Transit Routing, Karlsruhe Institute of Technology, Delling, Daniel; Pajor Thomas, Werneck Renato. https://www.microsoft.com/en-us/research/wp-content/uploads/2012/01/raptor_alenex.pdf
• Motis Project https://github.com/motis-project Felix Gündling, innovation in in-memory data stores for graphs
• Hannah Bast, PhD at Freiburg Universitat: Transfer Patterns and Scalable Transfer Patterns
• Patrick Brosi: Pfaedle and Line optimised shapes

Submodules maintained

• DMFR dataset reader: reads data from transitland-atlas into raw structs https://docs.rs/dmfr-dataset-reader/latest/dmfr_dataset_reader/
• Château: Associates feeds with operators and vise versa using depth first search in knowledge graph

Dependent binaries maintained

• Pfaedle-rs: Shape matching in rust using low memory.

Agency specific submodules

• Amtrak GTFS rt: Conversion of proprietary realtime data from amtrak's website into gtfs-rt.
• Chicago GTFS Rt: conversion of proprietary Chicago CTA data to GTFS realtime
• Rtc Québec GTFS RT: conversion of proprietary app RTC Nomade to GTFS realtime
• Via Rail GTFS RT: Conversion of Via Rail tracking to GTFS Realtime.

Install Dependencies

sudo apt install -y postgresql-common
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
sudo apt install libprotoc-dev protobuf-compiler build-essential gcc pkg-config libssl-dev unzip wget cmake openssl libpq-dev

install coin cbc integer linear programming solver.

sudo apt-get install coinor-cbc coinor-libcbc-dev

Install postgres

sudo apt install postgresql-18 postgresql-18-postgis-3 postgresql-contrib postgresql

You may also use an external database if you prefer.

Enable PostGis

CREATE EXTENSION postgis;

For Contributors

Good commit messages are required to contribute to this project.

Installation of Postgres

See https://www.postgresql.org/download

PostGIS is also required like

sudo apt install postgresql-16-postgis-3

See https://trac.osgeo.org/postgis/wiki/UsersWikiPostGIS3UbuntuPGSQLApt for more instructions

SQL notes

We've switched to diesel for our queries. Read the diesel documentation to learn how to use it. https://diesel.rs/guides/getting-started.html

Lib PQ is also required to install the diesel cli. Only postgres is required. Example

sudo apt-get install libpq-dev
cargo install diesel_cli --no-default-features --features postgres

Common Database debugging

Is Postgis not installing? This page may be helpful: https://trac.osgeo.org/postgis/wiki/UsersWikiPostGIS3UbuntuPGSQLApt

Updating transitland submodules

git submodule update --rebase --remote

Future potential submodule names

• Truffle reachability analysis
• Chinese plum (needs to pick better syonym for it)

OSM Station Association

The OSM Station Import system associates GTFS railway stops with OpenStreetMap stations. This provides additional metadata like multilingual names, UIC references, and station relationships.

Step 1: Setup configuration and dependent databases.

Ensure that Postgres, Elasticsearch, and etcd are all running services.

Then copy example.catenaryconfig.toml to catenaryconfig.toml and insert the URLs and passwords for the services you wish.

Step 2: Import OSM Stations

Obtain a pre-filtered PBF file containing railway stations (or use one from Geofabrik filtered with osmium):

cargo run --bin osmstationimport -- --file /path/to/railstations.osm.pbf

multiple imports can be done sequentially, and will overwrite old data.

cargo run --bin osmstationimport -- --file railstations-europe-latest.osm.pbf
cargo run --bin osmstationimport -- --file railstations-north-america-latest.osm.pbf
cargo run --bin osmstationimport -- --file railstations-asia-latest.osm.pbf

The importer: - Computes SHA256 hash to skip duplicate imports - Extracts rail, tram, and subway stations - Parses multilingual names (name:en, name:de, etc.) - Stores in gtfs.osm_stations table with spatial indexing

Step 3: Run Maple Import

When Maple processes GTFS feeds, it automatically matches stops to OSM stations for rail/tram/subway routes:

cargo run --bin maple

See the Maple readme for more information.

Step 4: Launch at least one instance of Alpenrose, the GTFS rt downloader.

cargo run --bin alpenrose

Step 5: Launch at least one copy of Aspen, the GTFS realtime data processor.

cargo run --bin aspen

Step 5b: Launch Aspen Leader, the coordinator process for Aspen.

cargo run --bin aspenleader

Step 6: Launch Birch, the HTTP API provider

cargo run --bin birch

Step 7: Launch Spruce, the Websocket data provider

cargo run --bin spruce

Routing Engine Setup (HIGHLY UNSTABLE, IN PROGRESS)

The routing engine provides point-to-point transit directions using three main microservices:

1. Avens: OSM preprocessor to build walking/biking graphs.
2. Gentian: Timetable compiler to create a zero-copy transit timetable.
3. Edelweiss: The routing execution engine.

Step 1: Preprocess OSM Graphs with Avens

Avens takes OpenStreetMap PBF files and compiles them into a serialized routing graph format with an R-Tree spatial index.

To compile a batch of regions using a configuration file (recommended for managing multiple regions):

cargo run --bin avens --release -- batch --config avens_regions.toml

Example avens_regions.toml:

[global]
tmp_dir = "/tmp/avens_raw"
output_dir = "/path/to/graphs"

[[regions]]
name = "germany"
url = "https://download.geofabrik.de/europe/germany-latest.osm.pbf"

[[regions]]
name = "switzerland"
url = "https://download.geofabrik.de/europe/switzerland-latest.osm.pbf"

Procedure for Managing and Updating OSM PBFs: To manage your region endpoints, simply update the [[regions]] list in your TOML config file. Re-running the batch command will fetch the latest full PBFs from the listed URLs (overwriting old temp files) and regenerate the routing graphs in the output_dir seamlessly.

Alternatively, you can pull directly from a URL and compile a single region:

cargo run --bin avens --release -- pull --url https://download.geofabrik.de/europe/germany-latest.osm.pbf --output-dir /path/to/graphs/germany

To extract a locally downloaded PBF file:

cargo run --bin avens --release -- extract --input /path/to/germany-latest.osm.pbf --output-dir /path/to/graphs/germany

You can view statistics about a compiled graph directory using:

cargo run --bin avens --release -- info --graph-dir /path/to/graphs/germany

Step 2: Compile Timetables with Gentian

Gentian builds a RAPTOR-compatible zero-copy transit routing timetable from data imported into PostgreSQL (e.g., via Maple). It integrates with the Avens graphs to accurately compute transfer footpaths.

cargo run --bin gentian --release -- --output /path/to/timetable.bin --avens-graphs /path/to/graphs

Step 3: Run the Edelweiss Routing Engine

Edelweiss provides the actual routing service via a Tarpc RPC server (listening on 0.0.0.0:9090). It uses the Avens graphs for OSM routing and dynamically loads timetables compiled by Gentian.

GRAPH_DIR=/path/to/graphs cargo run --bin edelweiss --release