OxiMedia

Pure Rust reconstruction of OpenCV + FFmpeg — A patent-free, memory-safe multimedia and computer vision framework.

Vision

OxiMedia is a clean room, Pure Rust reconstruction of both FFmpeg (multimedia processing) and OpenCV (computer vision) — unified in a single cohesive framework.

FFmpeg Domain

Codec encoding/decoding (AV1, VP9, VP8, Theora, Opus, Vorbis, FLAC, MP3), container muxing/demuxing (MP4, MKV, MPEG-TS, OGG), streaming protocols (HLS, DASH, RTMP, SRT, WebRTC, SMPTE 2110), transcoding pipelines, filter graphs (DAG-based), audio metering (EBU R128), loudness normalization, packaging (CMAF, DRM/CENC), and server-side media delivery.

OpenCV Domain

Computer vision (object detection, motion tracking, video enhancement, quality assessment), professional image I/O (DPX, OpenEXR, TIFF), video stabilization, scene analysis, shot detection, denoising (spatial/temporal/hybrid), camera calibration, color management (ICC, ACES, HDR), video scopes (waveform, vectorscope, histogram), and forensic analysis (ELA, PRNU, copy-move detection).

Design Principles

• Patent Freedom: Only royalty-free codecs (AV1, VP9, Opus, FLAC, and more)
• Memory Safety: Zero unsafe code, compile-time guarantees
• Async-First: Built on Tokio for massive concurrency
• Single Binary: No DLL dependencies, no system library requirements
• WASM Ready: Runs in browser without transcoding servers
• Sovereign: No C/Fortran dependencies in default features — 100% Pure Rust

FFmpeg + OpenCV, Reimagined

FFmpeg is the de facto standard for multimedia processing, but it is written in C with patent-encumbered codecs (H.264, H.265, AAC), chronic memory safety vulnerabilities, and notoriously complex build systems requiring dozens of system libraries.

OpenCV is the de facto standard for computer vision, but it depends on C++ with complex CMake builds, optional proprietary modules (CUDA, Intel IPP), and heavy system-level dependencies.

OxiMedia unifies both into a single Pure Rust framework with zero C/Fortran dependencies:

From the FFmpeg world: codec encode/decode, container mux/demux, streaming (HLS/DASH/RTMP/SRT/WebRTC), transcoding pipelines, filter graphs, audio processing, packaging, and media server.

From the OpenCV world: detection, tracking, stabilization, scene analysis, shot detection, denoising, calibration, image I/O (DPX/EXR/TIFF), color science, quality metrics (PSNR/SSIM/VMAF), and forensics.

One cargo add — no battling system library installations, no pkg-config, no LD_LIBRARY_PATH, no brew install ffmpeg opencv.

Project Scale

OxiMedia is a production-grade framework at v0.1.8 (active cycle, 2026-06-02):

Sovereign ML Pipelines (v0.1.7+)

OxiMedia 0.1.7 introduced the oximedia-ml crate — a typed ML pipeline layer built atop the Pure-Rust OxiONNX runtime. Inference is entirely opt-in; the default oximedia build still pulls in zero ONNX symbols and stays C/Fortran-free.

Available pipelines

Quick start

[dependencies]
oximedia = { version = "0.1.8", features = ["ml", "ml-scene-classifier", "ml-onnx"] }

```rust,ignore use oximedia::ml::pipelines::{SceneClassifier, SceneImage}; use oximedia::ml::{DeviceType, TypedPipeline};

let classifier = SceneClassifier::load("places365.onnx", DeviceType::auto())?; let image = SceneImage::new(rgb_bytes, 224, 224)?; for pred in classifier.run(image)? { println!("class {} -> {:.3}", pred.class_index, pred.score); }

### Device selection

`DeviceType::auto()` probes the strongest available backend once and
memoises the result: **CUDA → DirectML → WebGPU → CPU**. Each backend is a
feature flag (`cuda`, `directml`, `webgpu`); CPU is always available.
`cuda` is native-only; every other backend — including the default CPU path
— compiles on `wasm32-unknown-unknown`.

### CLI

The `oximedia ml` namespace ships three subcommands (honour `--json` for
machine-readable output):

```bash
oximedia ml list                  # enumerate built-in pipelines + model zoo
oximedia ml probe                 # report GPU backend availability
oximedia ml run --pipeline scene-classifier \
                --model places365.onnx \
                --input frame.png \
                --device auto \
                --top-k 5 \
                --dry-run

Downstream integrations

Several domain crates gain an onnx feature for an ML-backed fast path while keeping the Pure-Rust default intact: oximedia-scene (MlSceneEnricher), oximedia-shots (MlShotDetector), oximedia-caption-gen (CaptionEncoder), oximedia-recommend (EmbeddingExtractor), oximedia-mir (MusicTagger).

See docs/ml_guide.md for the full feature matrix, per-pipeline I/O contracts, device selection details, WASM support matrix, and roadmap.

What's New in v0.1.8

Active cycle — latest release 2026-06-02. Theme: Codec completeness, audio restoration, algorithmic depth, and entropy coding improvements.

• SILK encoder with NSQ noise-shaped quantisation (oximedia-audio): Real SILK encoder path with noise-shaped quantisation loop; 440 Hz sine round-trip SNR ≥ 6 dB verified.
• AV1 non-square TX block coefficient decoding fixed (oximedia-codec): CoeffBuffer::pos_to_rowcol now correctly handles non-square transform blocks, fixing a symbol-vs-position bug in AV1 entropy decoding EOB CDF paths.
• AAF binary serializer (oximedia-aaf): Full SMPTE ST 377-1 CFB + KLV binary serializer; 22 previously orphan AAF modules registered and wired into the workspace.
• NDI SpeedHQ Huffman entropy coding (oximedia-ndi): Real Huffman entropy coding for NDI SpeedHQ streams; 22 NDI orphan modules registered and wired.
• DRM software TPM 2.0 emulator + Secure Enclave emulator (oximedia-drm): Pure-Rust software TPM 2.0 emulator and Secure Enclave emulator — enables DRM key protection on platforms without hardware TPM/SE.
• Audio restoration: AR-LPC declick + Boll 1979 spectral subtraction + Wiener denoiser (oximedia-restore): Three new audio restoration algorithms: AR-LPC-based declicker, Boll 1979 spectral subtraction noise reducer, and a parametric Wiener filter denoiser.
• Y4M reader/writer + ITU-T P.910 SI/TI/motion metrics (oximedia-bench): Y4M (YUV4MPEG2) container reader/writer and full ITU-T P.910 Spatial Information, Temporal Information, and motion activity metrics.
• S3 multipart upload with retry + configurable parallelism (oximedia-server): S3 multipart upload with per-part retry logic and configurable upload parallelism for high-throughput media ingest.
• FLAC/Opus/MP3/Vorbis waveform decode wired (oximedia-clips): Clip waveform extraction now uses the real demuxer path for FLAC, Opus, MP3, and Vorbis — replacing stub silence.
• SRT ingest server wired to real SrtListener::accept (oximedia-net): SRT ingest now calls the real SrtListener::accept instead of the previous no-op stub, enabling live SRT stream ingestion.
• AutoCaptionPipeline example refreshed; ONNX/ML pipeline improvements (oximedia-ml, oximedia-caption-gen): AutoCaption example updated to current API; ML pipeline feature flags and ONNX runtime wiring improved across the workspace.
• AnalysisScale Half/Quarter + downsample_box_luma (oximedia-analysis): Configurable downscaling (Full/Half/Quarter) for the analysis pipeline; box-filter downsample; 5 new tests.
• rFFT phase correlation (oximedia-align): phase_correlate_1d now uses oxifft::rfft/irfft (N/2+1 bins, half the complex ops) matching the OxiFFT policy; 4 regression tests.
• DataCite 4.x + PBCore 2.1 + MigrationTriggerPolicy (oximedia-archive-pro): DataCite DOI metadata generation, PBCore metadata crosswalk, and automated format migration triggers; 18 new tests.
• batch_conform + ProxyDbExport / import_with_rebase (oximedia-proxy): Batch EDL conforming with merge strategies, proxy database export/import with root-prefix rebase; 9 new tests.
• SegmentPlan + encode_segments_parallel (oximedia-convert): Keyframe-boundary segment plan with rayon-parallel encode; codec-agnostic concat; 5 new tests.
• scale_tiled + scale_reference (oximedia-scaling): Cache-blocked tiled scaling (bit-exact vs reference), rayon par_iter over tiles; 7 new tests.
• Speech-clarity biquad DRC + SIMD contrast enhancement (oximedia-access): Real speech DSP (4th-order Butterworth 300–3400 Hz, downward DRC, peaking boost) and AVX2/NEON SIMD contrast enhancement with 256-entry gamma LUT.
• NSQ 440 Hz SNR fix (oximedia-codec): SILK LTP coarse-to-fine decimated pitch search, per-subframe contour RD, fractional-lag refinement, round-trip harness.
• Waves 1–20 complete, 100,278 tests passing (0 failures, 0 warnings — cargo nextest run --workspace --all-features).
• Zero clippy warnings workspace-wide; WASM check clean.

What's New in v0.1.5

Released 2026-04-21. Theme: Full ONNX Runtime integration via the Pure-Rust OxiONNX stack (previously slated for 0.3.0). All inference is feature-gated; the pure-Rust default build stays C/Fortran-free.

• Pure-Rust ONNX inference via OxiONNX: Workspace now depends on the full OxiONNX stack (oxionnx, oxionnx-ops, oxionnx-gpu, oxionnx-directml, oxionnx-proto) instead of the C++ ort runtime — preserves the COOLJAPAN Pure-Rust Policy.
• New oximedia-ml facade crate: Central ML layer exposing OnnxModel, ModelCache (concurrent LRU model cache with ~/.cache/oximedia/models/), TypedPipeline<In, Out> trait, DeviceType::auto() runtime probe, ImagePreprocessor (ImageNet normalize + letterbox + NCHW), and ModelZoo registry.
• Typed pipelines: SceneClassifier (ImageNet-style top-K classifier with softmax/argsort postprocessing) and ShotBoundaryDetector (TransNetV2-compatible sliding window with many-hot hard/soft cut outputs) — more pipelines (AutoCaption, AestheticScore, ObjectDetector, FaceEmbedder) queued for Wave 2.
• Facade ml feature: oximedia::ml module and prelude re-exports gated behind ml (off by default). Sub-features ml-scene-classifier, ml-shot-boundary, ml-onnx for granular opt-in; full feature picks them all up.
• 55 new tests in oximedia-ml: Covers preprocessing, cache eviction, pipeline contracts, and ModelInfo round-trips. Zero clippy warnings. Pure-Rust default build verified — no ONNX symbols linked unless onnx feature is enabled.

What's New in v0.1.4

Released 2026-04-20.

• MJPEG and APV end-to-end codec support: Full encode/decode pipelines for MJPEG (Motion JPEG) and APV (Advanced Professional Video), including correct MP4 and Matroska sample-entry wiring.
• JPEG encoder/decoder spec compliance: Rebuilt JPEG encoder/decoder to full JFIF/Exif spec compliance; PSNR improved from 6 dB to 32 dB at quality 85.
• AVI container muxer + demuxer: New oximedia-avi crate — pure-Rust AVI muxer and demuxer targeting MJPEG-only streams up to 1 GB (RIFF list size constraint).
• AJXL ISOBMFF animated encoder + streaming decoder iterator: Animated JPEG-XL sequences stored in ISOBMFF with a streaming Iterator-based decoder for low-memory playback.
• CLI MJPEG/APV support: oximedia-cli now accepts -c:v mjpeg and -c:v apv on all transcode/convert commands.
• WASM32 platform support: 5 additional crates (oximedia-codec, oximedia-container, oximedia-audio, oximedia-convert, oximedia-graphics) confirmed clean under wasm32-unknown-unknown.

Architecture

FFmpeg domain spans