DDIF: Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images

Arxiv Information Fusion

Abstract: The denoising diffusion model has received increasing attention in the field of image generation in recent years, thanks to its powerful generation capability. However, diffusion models should be deeply investigated in the field of multi-source image fusion, such as remote sensing pansharpening and multispectral and hyperspectral image fusion (MHIF). In this paper, we introduce a novel {supervised} diffusion model with two conditional modulation modules, specifically designed for the task of multi-source image fusion. These modules mainly consist of a coarse-grained style modulation (CSM) and a fine-grained wavelet modulation (FWM), which aim to disentangle coarse-grained style information and fine-grained frequency information, respectively, thereby generating competitive fused images. Moreover, some essential strategies for the training of the given diffusion model are well discussed, e.g., the selection of training objectives. The superiority of the proposed method is verified compared with recent state-of-the-art (SOTA) techniques by extensive experiments on two multi-source image fusion benchmarks, i.e., pansharpening and MHIF. In addition, sufficient discussions and ablation studies in the experiments are involved to demonstrate the effectiveness of our approach.

News: - 2024/4/21: A new paper about Neural Shrödinger Bridge Matching for Pansharpening is released on arxiv.🤗

• 2023/12/4：Code RELEASED!:fire:

• 2023/11/23: Code will be released soon!:fire:

Quick Overview

The code in this repo supports Pansharpening, Hyperspectral and multispectral image fusion.

Instructions

Dataset

In this office repo, you can find the Pansharpening dataset of WV3, GF2, and QB.

We follow the PSRT to implement Hyperspectral and multispectral image fusion. You can find the data we use in this repo.

Other instructions will come soon!

Citation

If you find our paper useful, please consider citing the following:

@article{DDIF,
title = {Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images},
journal = {Information Fusion},
volume = {104},
pages = {102158},
year = {2024},
issn = {1566-2535},
doi = {https://doi.org/10.1016/j.inffus.2023.102158},
url = {https://www.sciencedirect.com/science/article/pii/S1566253523004748},
author = {Zihan Cao and Shiqi Cao and Liang-Jian Deng and Xiao Wu and Junming Hou and Gemine Vivone},
}