arborium

Batteries-included tree-sitter grammar collection with HTML rendering and WASM support.

Features

• {{TOTAL_COUNT}} language grammars included out of the box
• {{PERMISSIVE_COUNT}} permissively licensed (MIT/Apache-2.0/CC0/Unlicense) grammars enabled by default
• WASM support with custom allocator fix
• Feature flags for fine-grained control over included languages

Usage

[dependencies]
arborium = "0.1"

By default, all permissively-licensed grammars are included. To select specific languages:

[dependencies]
arborium = { version = "0.1", default-features = false, features = ["lang-rust", "lang-javascript"] }

Feature Flags

Grammar Collections

Permissive Grammars ({{PERMISSIVE_COUNT}})

These grammars use permissive licenses (MIT, Apache-2.0, CC0, Unlicense) and are included by default.

{{PERMISSIVE_TABLE}}

GPL-Licensed Grammars ({{GPL_COUNT}})

These grammars are not included by default due to their copyleft license. Enabling them may have implications for your project's licensing.

{{GPL_TABLE}}

Sponsors

CI infrastructure generously provided by Depot.

License

This project is dual-licensed under MIT OR Apache-2.0.

The bundled grammar sources retain their original licenses - see LICENSES.md for details.

WASM Support

Arborium supports building for wasm32-unknown-unknown. This requires compiling C code (tree-sitter core and grammar parsers) to WebAssembly.

macOS

On macOS, the built-in Apple clang does not support the wasm32-unknown-unknown target. You need to install LLVM via Homebrew:

brew install llvm

Then ensure the Homebrew LLVM is in your PATH when building:

export PATH="$(brew --prefix llvm)/bin:$PATH"
cargo build --target wasm32-unknown-unknown

FAQ

Build fails with "No available targets are compatible with triple wasm32-unknown-unknown"

Error message:

error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'

Cause: You're using Apple's built-in clang, which doesn't include the WebAssembly backend.

Solution: Install LLVM via Homebrew and use it instead:

brew install llvm
export PATH="$(brew --prefix llvm)/bin:$PATH"
cargo build --target wasm32-unknown-unknown

You may want to add the PATH export to your shell profile (.zshrc, .bashrc, etc.) or use a tool like direnv to set it per-project.

Development

This project uses cargo xtask for development tasks. Run cargo xtask help to see available commands.

Repository Structure

arborium/
├── grammars/                    # Vendored grammar sources (Git-tracked)
│   └── tree-sitter-{lang}/
│       ├── grammar.js           # Grammar definition
│       ├── src/                 # Generated parser (parser.c, scanner.c)
│       └── queries/             # Highlight/injection queries
├── crates/
│   └── arborium-{lang}/         # Per-language Rust crates (Git-tracked)
│       ├── Cargo.toml
│       ├── build.rs
│       ├── src/lib.rs
│       ├── grammar-src/         # C sources copied from grammars/ (Git-tracked)
│       ├── info.toml            # Language metadata (Git-tracked)
│       └── sample.{ext}         # Sample code for tests/demo (Git-tracked)
├── demo/                        # WASM demo website
│   ├── template.html            # HTML template (Git-tracked)
│   ├── index.html               # Generated HTML (Git-ignored)
│   ├── pkg/                     # Generated WASM package (Git-ignored)
│   └── language-info.json       # Language display info (Git-tracked)
├── GRAMMARS.toml                # Master grammar registry (Git-tracked)
└── README.md                    # Generated from template (Git-tracked)

What's Stored in Git vs Generated

xtask Commands

cargo xtask regenerate [name]

Regenerates tree-sitter parsers from grammar.js files.

What it does: 1. Finds all grammars/tree-sitter-* directories with a grammar.js 2. Groups grammars by dependency level (e.g., C before C++, CSS before SCSS) 3. For each grammar: - Runs tree-sitter init --update if tree-sitter.json exists - Runs pnpm install if package.json exists and node_modules/ is missing - Runs tree-sitter generate to create/update src/parser.c 4. Cleans up generated files we don't need (bindings, etc.) 5. Copies C sources (parser.c, scanner.c, etc.) to crates/arborium-*/grammar-src/

When to run: After modifying a grammar.js or updating a grammar from upstream.

Optional filter: cargo xtask regenerate rust to regenerate only matching grammars.

cargo xtask vendor <name>

Updates a grammar from its upstream repository.

What it does: 1. Clones the latest version from the repo URL in GRAMMARS.toml 2. Copies essential files: grammar.js, LICENSE, src/, helper JS files 3. Preserves local queries/ directory (your highlight query customizations) 4. Updates the commit hash in GRAMMARS.toml

After vendoring: Run cargo xtask regenerate <name> then cargo build.

cargo xtask check-updates

Checks if any vendored grammars have newer versions available upstream.

Compares the commit hash in GRAMMARS.toml against the latest HEAD of each grammar's repository.

cargo xtask generate-crates

Generates arborium-* crates for all grammars.

Important: Skips crates that already exist (to preserve manual edits). Delete a crate directory to regenerate it.

What it generates: - Cargo.toml with correct dependencies - build.rs that compiles the C sources - src/lib.rs with language function and query exports

cargo xtask generate-demo

Generates the demo website's index.html from template.html.

What it does: 1. Reads language-info.json for language display names and icons 2. Fetches icons from Iconify API (cached in .icon-cache.json) 3. Reads app.js and styles.css and inlines them 4. Reads sample files from demo/examples/ directory 5. Generates index.html with all assets inlined

cargo xtask generate-readme

Generates README.md from README.template.md and GRAMMARS.toml.

Replaces placeholders like TOTAL_COUNT, PERMISSIVE_TABLE, etc. (wrapped in double curly braces).

cargo xtask serve-demo [options]

Builds and serves the WASM demo locally.

Options: - -a <address> - Bind address (default: 127.0.0.1) - -p <port> - Port (default: auto-select 8000-8010) - --dev - Fast dev build: -O1, no wasm-opt, fast compression

What it does: 1. Runs generate-demo to create index.html 2. Builds WASM with wasm-pack build --target web 3. Checks for env imports (these won't work in browsers) 4. Pre-compresses files with brotli and gzip 5. Starts HTTP server with compression support

Typical Workflows

Adding a New Grammar

# 1. Add entry to GRAMMARS.toml with repo URL and license
# 2. Vendor the grammar
cargo xtask vendor newlang

# 3. Regenerate parser
cargo xtask regenerate newlang

# 4. Generate the crate
cargo xtask generate-crates

# 5. Add to workspace in root Cargo.toml
# 6. Create info.toml with language metadata
# 7. Add sample file for tests/demo
# 8. Test
cargo test -p arborium-newlang

Updating a Grammar

# 1. Check for updates
cargo xtask check-updates

# 2. Vendor the update
cargo xtask vendor rust

# 3. Regenerate
cargo xtask regenerate rust

# 4. Test
cargo build && cargo test

Fixing Highlight Queries

Highlight queries live in grammars/tree-sitter-{lang}/queries/highlights.scm. Edit them directly - they won't be overwritten by vendor (queries are preserved).

After editing, rebuild the demo to test: cargo xtask serve-demo --dev