Token-Oriented Object Notation (TOON)

Token-Oriented Object Notation is a compact, human-readable encoding of the JSON data model that minimizes tokens and makes structure easy for models to follow. It's intended for LLM input as a drop-in, lossless representation of your existing JSON.

TOON combines YAML's indentation-based structure for nested objects with a CSV-style tabular layout for uniform arrays. TOON's sweet spot is uniform arrays of objects (multiple fields per row, same structure across items), achieving CSV-like compactness while adding explicit structure that helps LLMs parse and validate data reliably. For deeply nested or non-uniform data, JSON may be more efficient.

The similarity to CSV is intentional: CSV is simple and ubiquitous, and TOON aims to keep that familiarity while remaining a lossless, drop-in representation of JSON for Large Language Models.

Think of it as a translation layer: use JSON programmatically, and encode it as TOON for LLM input.

[!TIP] The TOON format is stable, but also an idea in progress. Nothing's set in stone – help shape where it goes by contributing to the spec or sharing feedback.

Table of Contents

• Why TOON?
• Key Features
• When Not to Use TOON
• Benchmarks
• Installation & Quick Start
• Playgrounds
• Editor Support
• CLI
• Format Overview
• Using TOON with LLMs
• Documentation
• Other Implementations
• 📋 Full Specification

Why TOON?

AI is becoming cheaper and more accessible, but larger context windows allow for larger data inputs as well. LLM tokens still cost money – and standard JSON is verbose and token-expensive:

{
  "context": {
    "task": "Our favorite hikes together",
    "location": "Boulder",
    "season": "spring_2025"
  },
  "friends": ["ana", "luis", "sam"],
  "hikes": [
    {
      "id": 1,
      "name": "Blue Lake Trail",
      "distanceKm": 7.5,
      "elevationGain": 320,
      "companion": "ana",
      "wasSunny": true
    },
    {
      "id": 2,
      "name": "Ridge Overlook",
      "distanceKm": 9.2,
      "elevationGain": 540,
      "companion": "luis",
      "wasSunny": false
    },
    {
      "id": 3,
      "name": "Wildflower Loop",
      "distanceKm": 5.1,
      "elevationGain": 180,
      "companion": "sam",
      "wasSunny": true
    }
  ]
}

YAML already conveys the same information with fewer tokens.

context:
  task: Our favorite hikes together
  location: Boulder
  season: spring_2025
friends:
  - ana
  - luis
  - sam
hikes:
  - id: 1
    name: Blue Lake Trail
    distanceKm: 7.5
    elevationGain: 320
    companion: ana
    wasSunny: true
  - id: 2
    name: Ridge Overlook
    distanceKm: 9.2
    elevationGain: 540
    companion: luis
    wasSunny: false
  - id: 3
    name: Wildflower Loop
    distanceKm: 5.1
    elevationGain: 180
    companion: sam
    wasSunny: true

TOON conveys the same information with even fewer tokens – combining YAML-like indentation with CSV-style tabular arrays:

context:
  task: Our favorite hikes together
  location: Boulder
  season: spring_2025
friends[3]: ana,luis,sam
hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}:
  1,Blue Lake Trail,7.5,320,ana,true
  2,Ridge Overlook,9.2,540,luis,false
  3,Wildflower Loop,5.1,180,sam,true

Key Features

• 📊 Token-Efficient & Accurate: TOON reaches 76.4% accuracy (vs JSON's 75.0%) while using ~40% fewer tokens in mixed-structure benchmarks across 4 models.
• 🔁 JSON Data Model: Encodes the same objects, arrays, and primitives as JSON with deterministic, lossless round-trips.
• 🛤️ LLM-Friendly Guardrails: Explicit [N] lengths and {fields} headers give models a clear schema to follow, improving parsing reliability.
• 📐 Minimal Syntax: Uses indentation instead of braces and minimizes quoting, giving YAML-like readability with CSV-style compactness.
• 🧺 Tabular Arrays: Uniform arrays of objects collapse into tables that declare fields once and stream row values line by line.
• 🌐 Multi-Language Ecosystem: Spec-driven implementations in TypeScript, Python, Go, Rust, .NET, and other languages.

What's New in v3.2

• Strict mode rejects duplicate sibling keys; with strict: false the last value wins.
• Malformed array headers ([03], [1] foo:, header/field delimiter mismatch) error in strict mode instead of silently degrading.
• Tabular form excludes arrays that contain an empty {} element – those fall back to the expanded list form.
• Nested arrays-of-objects can appear as list items via an explicit - [N]: header.

Media Type & File Extension

By convention, TOON files use the .toon extension and the provisional media type text/toon for HTTP and content-type–aware contexts. TOON documents are always UTF-8 encoded; the charset=utf-8 parameter may be specified but defaults to UTF-8 when omitted. See SPEC.md §17 for normative details.

When Not to Use TOON

TOON excels with uniform arrays of objects, but there are cases where other formats are better:

• Deeply nested or non-uniform structures (tabular eligibility ≈ 0%): JSON-compact often uses fewer tokens. Example: complex configuration objects with many nested levels.
• Semi-uniform arrays (~40–60% tabular eligibility): Token savings diminish. Prefer JSON if your pipelines already rely on it.
• Pure tabular data: CSV is smaller than TOON for flat tables. TOON adds minimal overhead (~5–10%) to provide structure (array length declarations, field headers, delimiter scoping) that improves LLM reliability.
• Latency-critical applications: If end-to-end response time is your top priority, benchmark on your exact setup. Some deployments (especially local/quantized models like Ollama) may process compact JSON faster despite TOON's lower token count. Measure TTFT, tokens/sec, and total time for both formats and use whichever is faster.

See benchmarks for concrete comparisons across different data structures.

Benchmarks

Benchmarks are organized into two tracks to ensure fair comparisons:

• Mixed-Structure Track: Datasets with nested or semi-uniform structures (TOON vs JSON, YAML, XML). CSV excluded as it cannot properly represent these structures.
• Flat-Only Track: Datasets with flat tabular structures where CSV is applicable (CSV vs TOON vs JSON, YAML, XML).

Retrieval Accuracy

Benchmarks test LLM comprehension across different input formats using 209 data retrieval questions on 4 models.

Show Dataset Catalog

Dataset Catalog

Dataset	Rows	Structure	CSV Support	Eligibility
Uniform employee records	100	uniform	✓	100%
E-commerce orders with nested structures	50	nested	✗	33%
Time-series analytics data	60	uniform	✓	100%
Top 100 GitHub repositories	100	uniform	✓	100%
Semi-uniform event logs	75	semi-uniform	✗	50%
Deeply nested configuration	11	deep	✗	0%
Valid complete dataset (control)	20	uniform	✓	100%
Array truncated: 3 rows removed from end	17	uniform	✓	100%
Extra rows added beyond declared length	23	uniform	✓	100%
Inconsistent field count (missing salary in row 10)	20	uniform	✓	100%
Missing required fields (no email in multiple rows)	20	uniform	✓	100%

Structure classes: - uniform: All objects have identical fields with primitive values - semi-uniform: Mix of uniform and non-uniform structures - nested: Objects with nested structures (nested objects or arrays) - deep: Highly nested with minimal tabular eligibility

CSV Support: ✓ (supported), ✗ (not supported – would require lossy flattening)

Eligibility: Percentage of arrays that qualify for TOON's tabular format (uniform objects with primitive values)

Efficiency Ranking (Accuracy per 1K Tokens)

Each format ranked by efficiency (accuracy percentage per 1,000 tokens):

TOON           ████████████████████   27.7 acc%/1K tok  │  76.4% acc  │  2,759 tokens
JSON compact   █████████████████░░░   23.7 acc%/1K tok  │  73.7% acc  │  3,104 tokens
YAML           ██████████████░░░░░░   19.9 acc%/1K tok  │  74.5% acc  │  3,749 tokens
JSON           ████████████░░░░░░░░   16.4 acc%/1K tok  │  75.0% acc  │  4,587 tokens
XML            ██████████░░░░░░░░░░   13.8 acc%/1K tok  │  72.1% acc  │  5,221 tokens

Efficiency score = (Accuracy % ÷ Tokens) × 1,000. Higher is better.

[!TIP] TOON achieves 76.4% accuracy (vs JSON's 75.0%) while using 39.9% fewer tokens.

Note on CSV: Excluded from ranking as it only supports 109 of 209 questions (flat tabular data only). While CSV is highly token-efficient for simple tabular data, it cannot represent nested structures that other formats handle.

Per-Model Accuracy

Accuracy across 4 LLMs on 209 data retrieval questions:

claude-haiku-4-5-20251001
→ TOON           ████████████░░░░░░░░    59.8% (125/209)
  JSON           ███████████░░░░░░░░░    57.4% (120/209)
  YAML           ███████████░░░░░░░░░    56.0% (117/209)
  XML            ███████████░░░░░░░░░    55.5% (116/209)
  JSON compact   ███████████░░░░░░░░░    55.0% (115/209)
  CSV            ██████████░░░░░░░░░░    50.5% (55/109)

gemini-3-flash-preview
  XML            ████████████████████    98.1% (205/209)
  JSON           ███████████████████░    97.1% (203/209)
  YAML           ███████████████████░    97.1% (203/209)
→ TOON           ███████████████████░    96.7% (202/209)
  JSON compact   ███████████████████░    96.7% (202/209)
  CSV            ███████████████████░    96.3% (105/109)

gpt-5-nano
→ TOON           ██████████████████░░    90.9% (190/209)
  JSON compact   ██████████████████░░    90.9% (190/209)
  JSON           ██████████████████░░    89.0% (186/209)
  CSV            ██████████████████░░    89.0% (97/109)
  YAML           █████████████████░░░    87.1% (182/209)
  XML            ████████████████░░░░    80.9% (169/209)

grok-4-1-fast-non-reasoning
→ TOON           ████████████░░░░░░░░    58.4% (122/209)
  YAML           ████████████░░░░░░░░    57.9% (121/209)
  JSON           ███████████░░░░░░░░░    56.5% (118/209)
  XML            ███████████░░░░░░░░░    54.1% (113/209)
  JSON compact   ██████████░░░░░░░░░░    52.2% (109/209)
  CSV            ██████████░░░░░░░░░░    51.4% (56/109)

[!TIP] TOON achieves 76.4% accuracy (vs JSON's 75.0%) while using 39.9% fewer tokens on these datasets.

Performance by dataset, model, and question type

Performance by Question Type

Question Type	TOON	JSON	YAML	JSON compact	XML	CSV
Field Retrieval	99.6%	99.3%	98.5%	98.5%	98.9%	100.0%
Aggregation	61.9%	61.9%	59.9%	58.3%	54.4%	50.9%
Filtering	56.8%	53.1%	56.3%	55.2%	51.6%	50.9%
Structure Awareness	89.0%	87.0%	84.0%	84.0%	81.0%	85.9%
Structural Validation	70.0%	60.0%	60.0%	55.0%	85.0%	80.0%

Performance by Dataset

Uniform employee records

Format	Accuracy	Tokens	Correct/Total
csv	73.2%	2,334	120/164
toon	73.2%	2,498	120/164
json-compact	73.8%	3,924	121/164
yaml	73.8%	4,959	121/164
json-pretty	73.8%	6,331	121/164
xml	74.4%	7,296	122/164

E-commerce orders with nested structures

Format	Accuracy	Tokens	Correct/Total
toon	82.3%	7,458	135/164
json-compact	78.7%	7,110	129/164
yaml	79.9%	8,755	131/164
json-pretty	79.3%	11,234	130/164
xml	77.4%	12,649	127/164

Time-series analytics data

Format	Accuracy	Tokens	Correct/Total
csv	75.0%	1,411	90/120
toon	78.3%	1,553	94/120
json-compact	74.2%	2,354	89/120
yaml	75.8%	2,954	91/120
json-pretty	75.0%	3,681	90/120
xml	72.5%	4,389	87/120

Top 100 GitHub repositories

Format	Accuracy	Tokens	Correct/Total
csv	65.9%	8,527	87/132
toon	66.7%	8,779	88/132
yaml	65.2%	13,141	86/132
json-compact	59.8%	11,464	79/132
json-pretty	63.6%	15,157	84/132
xml	56.1%	17,105	74/132

Semi-uniform event logs

Format	Accuracy	Tokens	Correct/Total
json-compact	68.3%	4,839