OxiCUDA

Pure Rust CUDA replacement -- cuBLAS, cuDNN, cuFFT, cuSPARSE, cuSOLVER, cuRAND and beyond in ~1.27M SLoC of safe Rust across 73 crates.

OxiCUDA replaces the entire NVIDIA CUDA Toolkit software stack with type-safe, memory-safe Rust code. The only runtime dependency is the NVIDIA driver (libcuda.so / nvcuda.dll); no CUDA SDK, no nvcc, no C/C++ toolchain is needed at build time. Optimized PTX assembly is generated directly from Rust data structures, and a built-in autotuner benchmarks kernel variants per GPU architecture to achieve near-peak throughput from Turing through Blackwell.

Architecture

+---------------------------------------------------------------+
|   SciRS2  |  OxiONNX  |  TrustformeRS  |  ToRSh              |
|   (Scientific Computing / ML / Inference Ecosystem)           |
+-------------------------------+-------------------------------+
                                |
+-------------------------------v-------------------------------+
|                         OxiCUDA                               |
|                     (Pure Rust GPU)                            |
|                                                               |
|  Vol.1 Foundation (4 crates)                                  |
|  +----------+ +--------+ +---------+ +---------+             |
|  | Driver   | | Memory | | Launch  | | Runtime |             |
|  +----------+ +--------+ +---------+ +---------+             |
|                                                               |
|  Vol.2 Codegen (2 crates)                                     |
|  +-----------+ +------------+                                 |
|  | PTX Gen   | | Autotune   |                                 |
|  +-----------+ +------------+                                 |
|                                                               |
|  Vol.3 Linear Algebra    Vol.4 Deep Learning                  |
|  +-------------+         +-------------+                      |
|  | BLAS        |         | DNN         |                      |
|  +-------------+         +-------------+                      |
|                                                               |
|  Vol.5 Scientific Computing (4 crates)                        |
|  +------+ +--------+ +--------+ +------+                     |
|  | FFT  | | Sparse | | Solver | | Rand |                     |
|  +------+ +--------+ +--------+ +------+                     |
|                                                               |
|  Vol.6 Signal    Vol.7 Comp.Graph  Vol.8 Training (2)         |
|  +---------+     +----------+      +-------+ +-------+        |
|  | Signal  |     | Graph    |      | Train | | Quant |        |
|  +---------+     +----------+      +-------+ +-------+        |
|                                                               |
|  Vol.9 Inference (3 crates)        Vol.10 RL                  |
|  +-------+ +------------+ +----+   +------+                   |
|  | Infer | | Dist-Infer | | LM |   |  RL  |                   |
|  +-------+ +------------+ +----+   +------+                   |
|                                                               |
|  Backends (7 crates)                                          |
|  +----------+ +--------+ +-------+ +--------+                 |
|  | backend  | | prims  | | Metal | | Vulkan |                 |
|  +----------+ +--------+ +-------+ +--------+                 |
|  +--------+ +-------+ +-----------+                           |
|  | WebGPU | | ROCm  | | LevelZero |                           |
|  +--------+ +-------+ +-----------+                           |
+-------------------------------+-------------------------------+
                                |
+-------------------------------v-------------------------------+
|              libcuda.so  (NVIDIA Driver, runtime only)        |
|              No SDK  /  No nvcc  /  No C Toolchain            |
+---------------------------------------------------------------+

Feature Highlights

Vol.1 -- Foundation (4 crates, 26,438 SLoC) - Dynamic driver loading via libloading -- zero build-time SDK dependency - DeviceBuffer<T> with Rust ownership semantics -- Send + Sync, RAII - Type-safe launch! macro with compile-time grid/block validation - CUDA Runtime API layer for high-level device management

Vol.2 -- PTX Codegen & Autotuner (2 crates, 47,429 SLoC) - Rust DSL that generates PTX IR covering SM 7.5 through SM 10.0 - Tensor Core support: WMMA, MMA, WGMMA instruction generation - Built-in autotuner with 3-tier dispatch (cached / tuned / default) - Disk-based PTX cache keyed by kernel hash + GPU architecture

Vol.3 -- BLAS (1 crate, 28,379 SLoC) - Full BLAS Level 1/2/3 (axpy, gemv, gemm, trsm, syrk, ...) - GEMM dispatch: SIMT, Tensor Core, Split-K paths - Batched GEMM: standard, strided, grouped - Precision coverage: F16, BF16, TF32, F32, F64, FP8 - Elementwise ops (relu, gelu, sigmoid, silu) and reductions (softmax, variance)

Vol.4 -- DNN (1 crate, 39,297 SLoC) - Convolution: implicit GEMM, im2col, Winograd 3x3, direct, fused Conv+BN+Act - FlashAttention forward/backward, PagedAttention, decode attention - MoE: top-k routing, token permutation, fused MoE kernel - Normalization: BatchNorm, LayerNorm, RMSNorm, GroupNorm - Pooling: max, average, adaptive, global - Resize: nearest, bilinear, bicubic - Quantization: FP8, INT8, block-scaled FP4

Vol.5 -- Scientific Computing (4 crates, 62,511 SLoC) - FFT: Stockham, radix-2/4/8, mixed-radix, Bluestein, C2C/R2C/C2R, 2D/3D - Sparse: CSR/CSC/COO/BSR/ELL, SpMV, SpMM, SpGEMM, SDDMM, ILU(0)/IC(0) - Solver: LU, QR, SVD, Cholesky, eigendecomp, CG, BiCGSTAB, GMRES - Rand: Philox, MRG32k3a, XORWOW, Sobol, uniform/normal/Poisson

Vol.6 -- Signal Processing (1 crate, 12,276 SLoC) - Audio: MFCC, STFT, Mel filterbank, spectral features - Image: Gaussian blur, Sobel edge detection, morphological ops - DCT: Types I-IV with fast algorithms - DWT: Haar, Daubechies wavelets - Filtering: IIR/FIR filters, Butterworth, Chebyshev - Correlation: cross-correlation, autocorrelation

Vol.7 -- Computation Graph (1 crate, 6,563 SLoC) - CUDA Graph capture API (StreamCapture, GraphCapture) - Execution plan with dependency-sorted node scheduling - Event-based inter-node synchronization - Sequential + parallel graph executors

Vol.8 -- GPU Training (2 crates, 13,832 SLoC) - Mixed precision training (AMP): FP16/BF16 + loss scaling - Gradient accumulation and clipping; EMA (exponential moving average) - LR schedulers: cosine, warmup, cyclic, polynomial - GPU-fused optimizers: Adam, AdamW, SGD, RMSProp, LAMB - Checkpointing (model save/load) - Quantization: INT8/INT4/FP8 weight quantization, block-scaled

Vol.9 -- Inference Engine (3 crates, 17,909 SLoC) - KV-cache with paged attention (PagedKvCache) and prefix caching - Speculative decoding - Distributed inference pipeline (tensor/pipeline parallelism) - LM inference: BPE tokenizer, vocabulary management, sampling strategies

Vol.10 -- Reinforcement Learning (1 crate, 11,280 SLoC) - Replay buffers: Uniform, Prioritized (PER), N-step - Policy distributions: Categorical, Gaussian (SAC reparameterization), Deterministic - Advantage estimators: GAE, TD(λ), V-trace, Retrace(λ) - Loss functions: PPO, DQN, Double-DQN, SAC, TD3 - Observation/reward normalization with Welford running stats - Environment abstractions: Env, VecEnv (auto-reset)

Backends (7 crates, 28,400 SLoC) - Backend trait abstraction for multi-GPU-runtime portability - CUB-equivalent GPU primitives (scan, reduce, sort, histogram) - Metal (macOS), Vulkan Compute, WebGPU, AMD ROCm, Intel oneAPI (LevelZero)

Pure Rust, Minimal Dependencies

OxiCUDA is built on a strict Pure Rust policy with minimal external dependencies. The entire codebase compiles with cargo build alone -- no C compiler, no Fortran runtime, no CUDA SDK, no nvcc, no pkg-config.

The only runtime requirement is the NVIDIA GPU driver (libcuda.so on Linux, nvcuda.dll on Windows). On macOS the crate compiles but returns UnsupportedPlatform at runtime.

Quick Start

use oxicuda::prelude::*;

fn main() -> Result<(), oxicuda::Error> {
    // Initialize driver and select GPU device
    let device = Device::get(0)?;
    let ctx = Context::new(device)?;
    let stream = Stream::new(&ctx)?;

    // Allocate device memory
    let mut d_a = DeviceBuffer::<f32>::zeroed(1024)?;
    let mut d_b = DeviceBuffer::<f32>::zeroed(1024)?;
    let mut d_c = DeviceBuffer::<f32>::zeroed(1024)?;

    // Copy host data to device
    d_a.copy_from_host(&host_a)?;
    d_b.copy_from_host(&host_b)?;

    // Launch a GEMM: C = alpha * A @ B + beta * C
    let handle = BlasHandle::new(&stream)?;
    handle.gemm(
        Transpose::None, Transpose::None,
        m, n, k,
        1.0f32,            // alpha
        &d_a, lda,
        &d_b, ldb,
        0.0f32,            // beta
        &mut d_c, ldc,
    )?;

    stream.synchronize()?;

    // Copy result back to host
    let mut result = vec![0.0f32; m * n];
    d_c.copy_to_host(&mut result)?;
    Ok(())
}

Crate Overview