Fluxon

An AI-native distributed data plane that supports high performance RPC, KV Cache, Message Queue, and File & Object Acceleration.

Add the project administrator on WeChat to join the open-source community group.

English | 中文 | Docs | 中文文档 |

Built on the unified Rust-based transport and caching foundation, Fluxon exposes three standardized interfaces that target the core bottlenecks in AI systems:

• KV/RPC (Unified key-value and RPC): Breaks data silos and enables efficient cross-process, cross-node reuse of inference-side KV Cache and latent cache
• MQ (Elastic message queue): Decouples system dependencies and supports elastic message transport across heterogeneous resource pools
• FS (S3-compatible file, object, and cache acceleration system): Unifies multi-form storage so one system can cache key-value, file, and object data, while supporting remote access, S3 forwarding, and large-scale cross-cluster migration for AI data and model files

🧭 Contents

• Benchmark
• Background
• Foundation Capabilities
• Interface Capabilities
• Runtime Requirements
• Quick Start
• Repository Structure
• Contributing
• Contributors
• Acknowledgements
• License
• Stargazers over time

📊 Benchmark

The benchmark section mainly covers the RPC, KV, and FS data planes, and the related scripts and configurations are primarily under fluxon_test_stack/.

Fluxon RPC Benchmark

The RPC benchmark mainly shows call latency and throughput across different message sizes and concurrency levels, to observe the stability and tail-latency behavior of the service-to-service call path.

Fluxon KV Benchmark

The TCP Benchmark shows that Fluxon outperforms MooncakeStore and Redis on the two read-heavy workloads Read-affinity and Read-Zipf. For put_only, the current primary constraint remains the inflight metadata deduplication path rather than Payload transport.

Fluxon KV can also use an owner's local SSD as a runtime backing layer for DRAM replicas. The chart below shows a single-node H100 SSD-pressure experiment measured through CUDA event completion. At c16, Fluxon's hit payload throughput for 4/8/16 MiB payloads was 3.83×/5.61×/6.73× that of the faster of the two Mooncake topologies.

See the Chinese deep dive, Fluxon KV SSD Storage: Using Local SSD as a Backing Layer for In-Memory Replicas, for the full setup, hit rates, and scope.

Fluxon FS Benchmark

Multi-node FS Python API

The benchmark results show that small-file reads and large-file writes already outperform Alluxio, large-file read performance is broadly on par, and small-file write performance still has further room to improve.

Single-node S3 API

In the reported single-node rclone v1.60.1 comparison with Alluxio S3 Proxy, FluxonFS led all 18 persisted-PUT and cold-read object-size/concurrency combinations. Hot-read gains were strongest for 4 KiB objects, while medium- and large-object sequential hot-read throughput was generally close.

Background

As GPU compute power continues to scale, bottlenecks in AI systems are expanding from individual operators into the data plane. Inference services need cross-node KV Cache reuse. Training pipelines need to pass intermediate state across heterogeneous resource pools. Model files and Checkpoint data need to move reliably between remote access paths and local caches.

Most existing systems, however, are still specialized components built for narrow scenarios, such as MooncakeStore for KV Cache. Many AI workloads still lack mature AI-native infrastructure components, so algorithm teams often assemble temporary data transfer modules just to validate ideas quickly. As model scale and cluster elasticity grow together, the cost of this patchwork data plane keeps expanding, consuming CPU, I/O, memory, and operational effort, and exposing seven critical engineering pain points:

• Poor generalization of domain-specific designs: specialized KV Cache systems bind cache semantics and RDMA transport to a narrow path, which makes them hard to carry over into more general data-plane scenarios
• Lack of unified resource governance: framework-level L2 and external L3 caches often live in the same host CPU memory, while L2 remains outside unified indexing and eviction control, increasing cache-crossing overhead
• Absence of a shared-memory fast path for local processes: many current data paths are organized around RDMA / TCP, so object handoff between colocated Workers still detours through the network protocol stack
• Lack of a dynamically elastic AI Infra communication plane: handoff across resource pools needs dynamic membership and asynchronous transfer, while fixed-member communication models amplify connection-management and recovery complexity
• Tight coupling between business processes and data-plane governance: when business processes start and stop dynamically while also contributing capacity, they trigger Rebalance churn and connection storms in the data plane
• Fragmented object lifecycle management: caches, messages, and files each maintain their own reference and eviction state, and those states easily fragment across business frameworks, cache layers, and transport layers
• Fragmented observability pipelines: cache hits, transport paths, and object materialization are scattered across separate systems, so performance debugging becomes an exercise in stitching clues together from multiple metric sets

🧱 Foundation Capabilities

• End-to-end Rust: consolidates connection handling, protocol encoding/decoding, state-machine progression, shared-memory management, and observability collection into Rust hot paths, reducing hot-path jitter from interpreted execution, cross-language boundaries, and uncontrolled copying
• Unified storage and transport: places storage and transport on one converged data plane, prioritizes the cross-process shared-memory fast path, and reduces fragmentation between object lifecycle management and transport behavior
• High-performance inter-node transport: prefers RDMA inside the cluster, supports automatic TCP fallback, and allows NICs to be enabled, disabled, and switched dynamically from the GUI, which lowers availability risk when one transport path degrades
• Automatic inter-node relay: supports automatic relay / forwarding across nodes and sub-clusters, reducing the integration cost of complex network topologies
• Global memory allocation and governance: uniformly manages global memory allocation, object lifecycles, capacity boundaries, and reclamation policies to avoid fragmentation and uncontrolled growth
• Unified role model: Master, Owner Client, and External Client cooperate in layers, organize control-plane and data-plane responsibilities into a scalable tree topology, and decouple business processes from data-plane governance to reduce Rebalance churn and connection storms
• Unified object interface: lets the system organize multi-field objects uniformly, balancing API flexibility, ease of use, and room for low-level optimization while keeping lifecycle state from scattering across layers
• Tensor-native zero-copy handoff path: facilitates the reuse of high-frequency tensor objects across caching and transport paths, eliminating the overhead of routing local process handoffs through the network stack
• Unified observability: uses the Prometheus protocol and Greptime to consolidate metric / trace / log, and includes a built-in GUI for cluster member state, log information, key metrics, and topology, which helps close observability gaps across systems
• Shared capabilities across all three interfaces: KV/RPC, MQ, and FS reuse the same caching, transport, lease, capacity-governance, and observability substrate, avoiding duplicated data-plane stacks for adjacent workloads

🔌 Interface Capabilities

Fluxon KV/RPC

Designed for world model inference caches, state sharing, service-to-service calls, and tensor object reuse. In scenarios such as multi-view latent-space prediction, state extrapolation, and prefix-cache reuse, Fluxon KV/RPC provides a more general AI data plane rather than a niche solution limited to a single KV Cache use case.

• Local cache replicas and eventually consistent read path: prioritizes local fast-path hits while synchronizing metadata asynchronously in the background
• Batched reclamation and hot-object management: advances invalid-object cleanup asynchronously through batch_delete, and combines it with TinyLFU to reuse hot objects more efficiently
• Simultaneous control over L2 and L3 in AI workloads: keeps global data objects indexed, discoverable, and reusable, reducing redundant memory waste from duplicate residency across cache tiers
• KV and RPC synergy: the same parameter organization, caching, and communication foundation serves both state storage and service-to-service calls

Fluxon MQ

Designed for heterogeneous training, data-processing pipelines, and intermediate-state handoff across resource pools. When the Producer side and Consumer side are split across different machines, different resource pools, or even different sub-clusters, Fluxon MQ consolidates message retention, capacity governance, and cross-cluster placement into one unified messaging layer.

• Lease-based retention semantics: binds message retention to the channel, ensuring data has bounded-time reliable retention before actual consumption
• channel-level prefix statistics and capacity governance: continuously tracks message counts and capacity usage boundaries for scaling and traffic control
• Cross-cluster load-aware placement: uses Consumer-side location to decide Payload placement, shortening prefetch paths and stabilizing throughput
• Co-designed with KV: message shells and member metadata stay on the control plane, while large Payload objects stay on the FluxonKV data plane, avoiding the need to build a second large-object transport stack

Fluxon FS

Fluxon FS is a high-performance, S3-compatible file and object cache for AI data and model files. It supports read/write acceleration, remote access, S3 forwarding, cache hits, and large-scale cross-cluster migration. In workloads with high-resolution video, trajectory samples, Checkpoint data, and other large file objects, Fluxon FS unifies these complex data flow and acceleration demands into a single data plane.

• Unified caching system: directly reuses FluxonKV/RPC caching and communication capabilities, splits files into KeyValue shards, and lets one system support accelerated reads and writes for key-value, file, and object caching
• S3 forwarding access: supports object-storage access and forwarding for AI data and model files
• Transparent Python file semantics: preserves the upper-layer open() / read() / write() experience as much as possible while reducing system-call and cross-process overhead
• Specialized optimization for small-file / large-file reads and writes: optimizes concurrency and transport paths by file granularity and read / write path to improve bandwidth utilization and overall throughput
• Large-scale cross-cluster migration: supports PB-scale data migration and keeps caching, transport, and failure recovery in one unified path

🧰 Runtime Requirements

For Quick Start (Docker):

• Docker installed
• The Quick Start image bundles the middleware required by the demo flows

For production deployment or building from source:

• OS: Linux only
• Python: >= 3.10
• Rust: Toolchain pinned to 1.93.0; see [fluxon_rs/rust-toolchai