github.com/fla-org/flash-linear-attention @v0.5.1

💥 Flash Linear Attention brings together hardware-efficient building blocks, training-ready layers, and components for modern sequence models, spanning linear attention, sparse attention, state space models, and hybrid LLM architectures. All implementations are platform-agnostic and verified on NVIDIA, AMD, and Intel hardware. Pull requests are welcome!

• News
• Models
• Installation
• Usage
• Token Mixing
• Fused Modules
• Generation
• Hybrid Models
• Training
• Evaluation
• Benchmarks
• Citation
• Star History
• Acknowledgements

News

• [2026-06] 🔭 Add Parallax implementation to fla (paper) — parameterized local linear attention: softmax attention with a learned first-order correction from a secondary query-side stream.
• [2026-06] 🧱 Add Wall attention implementation to fla (blog) — full softmax attention with a learned per-channel multiplicative decay, a RoPE-free positional encoding from Tilde Research.
• [2026-05] 🚪 Add Gated DeltaNet 2 (GDN-2) implementation to fla (paper) — decouples erase and write gates into independent channel-wise gates on top of KDA.
• [2026-05] 🦅 Add Raven implementation to fla (repo).
• [2026-05] 🚀 Add YOCO (You Only Cache Once) implementation to fla.
• [2026-05] ⚡ Add fused AttnRes support to fla (paper).
• [2026-04] 🐍 Add Mamba3 implementation to fla (paper).
• [2026-04] 🧱 Add MoBA (Mixture of Block Attention) implementation to fla, with FlashMoBA backend support.
• [2026-04] 🧱 Add TileLang backend support for selected kernels.
• [2026-04] 🎯 Add GPT-OSS-style attention sink support to fla's attention kernels.
• [2026-03] 🚀 Add Context Parallel support for KDA and GDN, enabling efficient distributed training across sequence dimension.
• [2025-10] 🌘 Add Kimi Delta Attention (KDA) implementation to fla (paper).
• [2025-09] 🌲 Add DeltaFormer implementation to fla (paper).
• [2025-09] 🐻 Thrilled to announce that GDN has been integrated into Qwen3-Next. Check out their blog post for more info!
• [2025-08] 🌲 Add Log-Linear Attention implementation to fla (paper).
• [2025-08] 🎓 Add MoM implementation to fla (paper).
• [2025-07] 🐳 Add MLA implementation to fla (paper).
• [2025-07] 🛣️ Add PaTH Attention implementation to fla (paper).

Older news

• [2025-06] 🎉 Add MesaNet implementation to fla (paper).
• [2025-06] 🐍 Add Comba implementation to fla (paper).
• [2025-05] 🎉 Add Rodimus* implementation to fla (paper).
• [2025-04] 🎉 Add DeltaProduct implementation to fla (paper).
• [2025-04] 🎉 Add FoX implementation to fla (paper).
• [2025-03] ~~We have changed the default initializer_range to the magic 🐳 0.006~~ The initializer_range was rolled back to the default value of 0.02. For actual training, we recommend trying both.
• [2025-02] 🐳 Add NSA implementations to fla. See kernels here.
• [2025-01] 🔥 We are migrating to torchtitan-based training framework. Check out the flame repo for more details.
• [2025-01] 🦅 Add RWKV7 implementations (both kernels and models) to fla.
• [2024-12] Add flash-bidirectional-attention to fla-org (repo).
• [2024-12] 🎉 Add Gated DeltaNet implementation to fla (paper).
• [2024-12] 🚀 fla now officially supports kernels with variable-length inputs.
• [2024-11] The inputs are now switched from head-first to seq-first format.
• [2024-11] 💥 fla now provides a flexible way for training hybrid models.
• [2024-10] 🔥 Announcing flame, a minimal and scalable framework for training fla models. Check out the details here.
• [2024-09] fla now includes a fused linear and cross-entropy layer, significantly reducing memory usage during training.
• [2024-09] 🎉 Add GSA implementation to fla (paper).
• [2024-05] 🎉 Add DeltaNet implementation to fla (paper).
• [2024-05] 💥 fla v0.1: a variety of subquadratic kernels/layers/models integrated (RetNet/GLA/Mamba/HGRN/HGRN2/RWKV6, etc., see Models).
• [2023-12] 💥 Launch fla, offering a collection of implementations for state-of-the-art linear attention models.

Models