Framework Benchmarks

Test each framework for it's performance, particularly common Lighthouse and CWV metrics as applications scale

Important: This is not a measure of "is framework x better than y". There are many tradeoffs to weigh when choosing the best framework for you - such as DX, features, familiarity, ecosystem, documentation, etc. These benchmarks only show a part of the picture.

Goals

The goal for this project are to understand the performance tradeoffs of popular frameworks in real world-ish scenarios. We want to assume non trivial codebases and imperfect code and conditions, so to see how each framework holds up to real world scenarios and scale.

We are intentionally not focused on client side rendering performance, for that please use Stefan Krause's great js-framework-benchmark

Status

This project is in initial development. Do not put too much weight on these current results, there is still much more to do to ensure accuracy, consistency, and fairness.

Contributions are welcome!

How it works

Methodology

We created a basic starting point for each framework in the frameworks/ folder using each framework's suggested starter/cli.

We then create basic example components and use Mitosis to compile them to best-effort idiomatic code for each framework. This will never be perfectly optimized code

We then build and serve each project, and run Lighthouse on each project with puppeteer, including with emulation of slow devices and networks (aka includes CPU and network throttling), and measure:

• FCP: First Contentful Paint (lower is better)
• LCP: Largest Contentful Paint (lower is better)
• TBT: Total Blocking Time (lower is better)
• TTI: Time to Interactive (lower is better)
• Score: Lighthouse Performance Score (higher is better)
• Eager JS Kib: the KiB of JS that is eagerly downloaded and executed from <script> tags for the initial page load. This is the actual size transferred over the network, including compression (lower is better)
• Total KiB: the total KiB transferred with the given page, including HTML, CSS, prefetched resources, etc. Also known as the "total byte weight". This is the actual size transferred over the network, including compression (lower is better)

We take the median of multiple runs, and sort the results by TTI, ascending

We are also experimenting with looking at other metrics, such as build times

The Frameworks

Alphabetically:

• astro - generated via their official CLI, with Solid for the interactive parts. Source
• fresh - generated via their official CLI. Source
• gatsby - contributed by the Gatsby team. Source
• hydrogen - generated via their official CLI. Source
• lit - generated via their official starter. Source
• marko - generated via their official CLI. Source
• next - generated via their official CLI. Source
• nuxt2 - generated via their official CLI. Source
• nuxt3 - generated via their official CLI (in beta). Source
• qwik - generated with Qwik City (meta framework). Source
• react - generated from create-react-app with react-router-dom added for routing. Source
• react-ssr-node - Ultra simple Node server to server-side render react. Source
• react-ssr-deno - Ultra simple Deno server to server-side render react. Source
• react-ssr-bun - Ultra simple Bun server to server-side render react. Source
• preact-ssr-node - Ultra simple Node server to server-side render preact. Source
• remix - generated from create-remix. Currently excluded from Lighthouse tests by request from the team, from concerns about the code being too non-idiomatic. Source
• solid - generated with Solid Start (meta framework). Source
• svelte - generated with Svelte Kit (meta framework). Source
• vue3 - generated via their official CLI, with routing. Source

Sample output

Important: This project is still in initial development. Do not put too much weight on these current results, there is still much more to do to ensure accuracy, consistency, and fairness.

Jump to:

• Dashboard
• Todo App
• Hello World
• SSR times
• SSR throughput (req/second)
• React SSR throughput
• Build times

Dashboard:

A more feature rich app for displaying table data, sorting, filtering, etc. Uses more JS, more like a median website. Source

┌─────────┬────────────┬──────────┬──────────┬──────────┬──────────┬───────┬──────────────┬───────────┐
│ (index) │    name    │   TTI    │   FCP    │   LCP    │   TBT    │ Score │ Eager JS KiB │ Total KiB │
├─────────┼────────────┼──────────┼──────────┼──────────┼──────────┼───────┼──────────────┼───────────┤
│    0    │   'qwik'   │ '0.6 s'  │ '0.6 s'  │ '1.5 s'  │  '0 ms'  │  100  │      2       │    38     │
│    1    │  'react'   │ '0.8 s'  │ '0.8 s'  │ '2.4 s'  │  '0 ms'  │  98   │     187      │    199    │
│    2    │  'gatsby'  │ '0.8 s'  │ '0.8 s'  │ '1.4 s'  │  '0 ms'  │  100  │      82      │    87     │
│    3    │   'lit'    │ '0.8 s'  │ '0.8 s'  │ '1.1 s'  │  '0 ms'  │  100  │      23      │    25     │
│    4    │  'solid'   │ '0.9 s'  │ '0.6 s'  │ '1.3 s'  │  '0 ms'  │  85   │      24      │    28     │
│    5    │  'astro'   │ '0.9 s'  │ '0.9 s'  │ '1.1 s'  │  '0 ms'  │  100  │      15      │    35     │
│    6    │  'marko'   │ '1.0 s'  │ '0.8 s'  │ '0.9 s'  │ '10 ms'  │  100  │      24      │    33     │
│    7    │  'fresh'   │ '1.3 s'  │ '1.3 s'  │ '1.5 s'  │  '0 ms'  │  100  │      17      │    46     │
│    8    │   'next'   │ '1.6 s'  │ '0.6 s'  │ '1.2 s'  │ '10 ms'  │  100  │      91      │    103    │
│    9    │  'svelte'  │ '1.6 s'  │ '1.6 s'  │ '1.7 s'  │  '0 ms'  │  99   │      29      │    35     │
│   10    │ 'angular'  │ '1.7 s'  │ '1.5 s'  │ '1.5 s'  │ '150 ms' │  98   │      86      │    88     │
│   11    │  'nuxt3'   │ '1.7 s'  │ '1.7 s'  │ '1.7 s'  │  '0 ms'  │  99   │      59      │    65     │
│   12    │   'vue3'   │ '1.8 s'  │ '1.2 s'  │ '2.1 s'  │  '0 ms'  │  94   │      41      │    50     │
│   13    │  'nuxt2'   │ '2.1 s'  │ '1.2 s'  │ '2.1 s'  │ '70 ms'  │  98   │     106      │    118    │
└─────────┴────────────┴──────────┴──────────┴──────────┴──────────┴───────┴──────────────┴───────────┘

Todo App:

A very simple/trivial interactive Todo app. Source

Ordered by TTI, ascending:

┌─────────┬────────────┬─────────┬─────────┬─────────┬──────────┬───────┬──────────────┬───────────┐
│ (index) │    name    │   TTI   │   FCP   │   LCP   │   TBT    │ Score │ Eager JS KiB │ Total KiB │
├─────────┼────────────┼─────────┼─────────┼─────────┼──────────┼───────┼──────────────┼───────────┤
│    0    │  'astro'   │ '0.6 s' │ '0.6 s' │ '0.6 s' │  '0 ms'  │  100  │      20      │    32     │
│    1    │   'qwik'   │ '0.7 s' │ '0.7 s' │ '1.2 s' │  '0 ms'  │  100  │      2       │    25     │
│    2    │  'react'   │ '0.8 s' │ '0.8 s' │ '2.2 s' │  '0 ms'  │  99   │     159      │    171    │
│    3    │  'fresh'   │ '0.8 s' │ '0.8 s' │ '0.9 s' │  '0 ms'  │  100  │      9       │    37     │
│    4    │   'lit'    │ '0.8 s' │ '0.8 s' │ '1.1 s' │  '0 ms'  │  100  │      16      │    18     │
│    5    │  'solid'   │ '1.0 s' │ '0.7 s' │ '1.3 s' │ '40 ms'  │  86   │      17      │    19     │
│    6    │  'marko'   │ '1.1 s' │ '0.8 s' │ '0.9 s' │ '10 ms'  │  100  │      17      │    23     │
│    7    │   'vue3'   │ '1.2 s' │ '1.2 s' │ '1.8 s' │ '10 ms'  │  99   │      33      │    41     │
│    8    │  'svelte'  │ '1.5 s' │ '1.5 s' │ '1.5 s' │  '0 ms'  │  100  │      19      │    24     │
│    9    │  'nuxt3'   │ '1.5 s' │ '1.5 s' │ '1.7 s' │  '0 ms'  │  99   │      50      │    55     │
│   10    │  'gatsby'  │ '1.6 s' │ '0.8 s' │ '1.1 s' │  '0 ms'  │  100  │      70      │    75     │
│   11    │  'nuxt2'   │ '1.6 s' │ '1.0 s' │ '1.0 s' │ '40 ms'  │  100  │      95      │    106    │
│   12    │ 'angular'  │ '1.6 s' │ '1.5 s' │ '1.6 s' │ '30 ms'  │  99   │      72      │    74     │
│   13    │   'next'   │ '2.2 s' │ '0.7 s' │ '0.8 s' │ '110 ms' │  99   │      83      │    94     │
└─────────┴────────────┴─────────┴─────────┴─────────┴──────────┴───────┴──────────────┴───────────┘

Hello World:

Just a few links and <h1>Hello World</h1>. Source

Ordered by TTI, ascending:

┌─────────┬────────────┬─────────┬─────────┬─────────┬──────────┬───────┬──────────────┬───────────┐
│ (index) │    name    │   TTI   │   FCP   │   LCP   │   TBT    │ Score │ Eager JS KiB │ Total KiB │
├─────────┼────────────┼─────────┼─────────┼─────────┼──────────┼───────┼──────────────┼───────────┤
│    0    │   'qwik'   │ '0.7 s' │ '0.7 s' │ '0.7 s' │  '0 ms'  │  100  │      0       │     4     │
│    1    │  'astro'   │ '0.7 s' │ '0.7 s' │ '0.7 s' │  '0 ms'  │  100  │      0       │     9     │
│    2    │  'react'   │ '0.8 s' │ '0.8 s' │ '2.2 s' │  '0 ms'  │  99   │     154      │    166    │
│    3    │  'fresh'   │ '0.8 s' │ '0.8 s' │ '0.8 s' │  '0 ms'  │  100  │      0       │    27     │
│    4    │  'marko'   │ '0.8 s' │ '0.8 s' │ '1.1 s' │  '0 ms'  │  100  │      15      │    21     │
│    5    │   'lit'    │ '0.8 s' │ '0.6 s' │ '0.9 s' │ '10 ms'  │  100  │      15      │    16     │
│    6    │  'solid'   │ '0.9 s' │ '0.9 s' │ '1.1 s' │  '0 ms'  │  100  │      16      │    18     │
│    7    │   'vue3'   │ '1.2 s' │ '1.2 s' │ '1.5 s' │  '0 ms'  │  100  │      31      │    38     │
│    8    │  'gatsby'  │ '1.4 s' │ '0.6 s' │ '1.0 s' │  '0 ms'  │  100  │      69      │    73     │
│    9    │  'svelte'  │ '1.5 s' │ '1.5 s' │ '1.5 s' │  '0 ms'  │  100  │      18      │    22     │
│   10    │  'nuxt2'   │ '1.5 s' │ '0.9 s' │ '0.9 s' │ '120 ms' │  99   │      93      │    103    │
│   11    │  'nuxt3'   │ '1.5 s' │ '1.5 s' │ '1.7 s' │  '0 ms'  │  99   │      50      │    57     │
│   12    │ 'angular'  │ '1.7 s' │ '1.5 s' │ '1.5 s' │ '150 ms' │  98   │      72      │    74     │
│   13    │ 'hydrogen' │ '1.8 s' │ '0.6 s' │ '1.6 s' │ '30 ms'  │  91   │     160      │    172    │
│   14    │   'next'   │ '2.1 s' │ '0.7 s' │ '0.8 s' │ '40 ms'  │  100  │      82      │    93     │
└─────────┴────────────┴─────────┴─────────┴─────────┴──────────┴───────┴──────────────┴───────────┘

SSR times:

Time it took to server-side render the /dashboard page in milliseconds. Smaller numbers are better.

┌─────────┬────────────┬─────┬─────┬─────┬────────┬─────────┐
│ (index) │    name    │ 1%  │ 50% │ 99% │  Avg   │ Std Dev │
├─────────┼────────────┼─────┼─────┼─────┼────────┼─────────┤
│    0    │  'marko'   │  1  │  1  │  4  │  1.29  │   0.8   │
│    1    │  'fresh'   │  4  │  4  │  6  │  4.19  │  0.88   │
│    2    │ 'hydrogen' │  4  │  5  │ 14  │  5.63  │  4.39   │
│    3    │  'svelte'  │  6  │  7  │ 18  │  8.17  │  3.12   │
│    4    │  'solid'   │  6  │  8  │ 22  │  8.77  │   3.8   │
│    5    │  'nuxt3'   │ 19  │ 25  │ 68  │ 29.49  │  18.09  │
│    6    │  'nuxt2'   │ 11  │ 14  │ 32  │ 15.04  │  4.89   │
│    7    │  'astro'   │ 11  │ 14  │ 38  │ 16.56  │  5.98   │
│    8    │  'remix'   │ 12  │ 18  │ 62  │  20.3  │  8.87   │
│    9    │  'gatsby'  │ 27  │ 33  │ 103 │  36.9  │  12.86  │
│   10    │   'next'   │ 35  │ 40  │ 113 │ 46.96  │  19.06  │
│   11    │ 'angular'  │ 113 │ 127 │ 400 │ 141.73 │  47.64  │
└─────────┴────────────┴─────┴─────┴─────┴────────┴─────────┘

SSR throughput (req/second):

SSR throughput of the dashboard page, measured by autocannon, sorted by 99% percentile descending. Larger numbers are better.

``` ┌─────────┬───────────────────┬──────┬──────┬──────┬─────────┬─────────┐ │ (index) │ name │ 1% │ 50% │ 99% │ Avg │ Std Dev │ ├─────────┼───────────────────┼──────┼──────┼──────┼─────────┼─────────┤ │ 0 │ 'marko' │ 3677 │ 5411 │ 5951 │ 5263.1 │ 702.18 │ │ 1 │ 'fresh' │ 1748 │ 2057 │ 2123 │ 1997.73 │ 117.92 │ │ 2 │ 'preact-ssr-node' │ 1755 │ 2065 │ 2185 │ 2052.6 │ 115.11 │ │ 3 │ 'hydrogen' │ 843 │ 1653 │ 1807 │ 1528.1 │ 290.27 │ │ 4 │ 'svelte' │ 536 │ 820 │ 1107 │ 795.3 │ 182.24 │ │ 5 │ 'solid' │ 534 │ 842 │ 1019 │ 830.64 │ 138.46 │ │ 6 │ 'astro'