[CVPR' 2025] JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration

<a href="https://lyl1015.github.io/">Yunlong Lin</a><sup>1*♣</sup>, 
<a href="https://github.com/LYL1015/JarvisIR">Zixu Lin</a><sup>1*♣</sup>, 
<a href="https://haoyuchen.com/">Haoyu Chen</a><sup>2*</sup>, 
<a href="https://paulpanwang.github.io/">Panwang Pan</a><sup>3*</sup>, 
<a href="https://chenxinli001.github.io/">Chenxin Li</a><sup>6</sup>, 
<a href="https://ephemeral182.github.io/">Sixiang Chen</a><sup>2</sup>, 
<a href="https://kairunwen.github.io/">Kairun Wen</a><sup>1</sup>, 
<a href="https://jinyeying.github.io/">Yeying Jin</a><sup>4</sup>, 
<a href="https://fenglinglwb.github.io/">Wenbo Li</a><sup>5†</sup>, 
<a href="https://scholar.google.com/citations?user=k5hVBfMAAAAJ&hl=zh-CN">Xinghao Ding</a><sup>1†</sup>













<sup>1</sup>Xiamen University, <sup>2</sup>The Hong Kong University of Science and Technology (Guangzhou), <sup>3</sup>Bytedance's Pico, <sup>4</sup>Tencent, <sup>5</sup>Huawei Noah's Ark Lab, <sup>6</sup>The Chinese University of Hong Kong

Accepted by CVPR 2025

JarvisIR Gradio Demo: Showcasing image restoration capabilities under various adverse weather conditions

:postbox: Updates

• 2025.10.1: Release CleanBench-Test (Paper test)🚀, please download for reproducibility!
• 2025.8.21: Release CleanBench-Real-80K (train and test)🚀, please download for training and evaluation! This test set contains a wider variety of degradation types and scenarios. It differs from the test set used in the paper.
• 2025.8.19: Release mrrhf training!
• 2025.7.6: Release degradation synthesis codes!
• 2025.6.26: Released inference code!
• 2025.6.17: Released 中文知乎解读 and JarvisIR：VLM掌舵,为自动驾驶装上“火眼金睛”,不惧恶劣天气 introducing JarvisIR! 📝
• 2025.6.13: Released Model weights and Huggingface Online demo 🤗 🚀 ✨.
• 2025.6.9: Released Gradio demo, restoration tools and SFT training code.
• 2025.4.8: This repo is created.

🧭 Navigation

• Overview
• Getting Started
• Gradio Demo
• SFT Training
• MRRHF Training
• Inference
• Degradation Synthesis
• Restoration Tools
• Acknowledgements

:diamonds: Overview

JarvisIR (CVPR 2025) is a VLM-powered agent designed to tackle the challenges of vision-centric perception systems under unpredictable and coupled weather degradations. It leverages the VLM as a controller to manage multiple expert restoration models, enabling robust and autonomous operation in real-world conditions. JarvisIR employs a novel two-stage framework consisting of supervised fine-tuning and human feedback alignment, allowing it to effectively fine-tune on large-scale real-world data in an unsupervised manner. Supported by CleanBench, a comprehensive dataset with 150K synthetic and 80K real instruction-response pairs, JarvisIR demonstrates superior decision-making and restoration capabilities, achieving a 50% improvement in the average of all perception metrics on CleanBench-Real.

:computer: Getting Started

For gradio demo runing, please follow:

• Gradio Demo

For inference and model usage, please follow:

• Inference Code

For image degradation data synthesis, please refer to:

• Degradation Generator

For sft training and environment setup preparation, please follow:

• SFT Training

For mrrhf training, please follow:

• MRRHF Training

:toolbox: Expert Models

JarvisIR integrates multiple expert restoration models to handle various types of image degradation. To test the performance of individual expert models, please refer to the instructions and scripts provided in ./package/agent_tools/.

:circus_tent: Checklist

• [x] Release inference code and gradio demo
• [x] Release sft training
• [x] Release Inference code
• [x] Release huggingFace online demo
• [x] Release degradation synthesis code
• [x] Release mrrhf training code
• [x] Release CleanBench-Real-80k dataset

:pray: Acknowledgements

We would like to express our gratitude to HuggingGPT, XTuner, IQA-PyTorch, and RRHF for their valuable open-source contributions which have provided important technical references for our work.

:love_you_gesture: Citation

@inproceedings{jarvisir2025,
  title={JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restoration},
  author={Lin, Yunlong and Lin, Zixu and Chen, Haoyu and Pan, Panwang and Li, Chenxin and Chen, Sixiang and Kairun, Wen and Jin, Yeying and Li, Wenbo and Ding, Xinghao},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2025}
}