github.com/ViTAE-Transformer/MTP @main
Chat with this repoWhat it actually does AI analysis from the code graph — generated when you open this
README
MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining
Di Wang, Jing Zhang, Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao and Liangpei Zhang
Update | Overview | Datasets and Models | Usage | Statement
🚩 Current applications
Remote Sensing Related Works: Please see Remote Sensing;
Remote Sensing Supervised Pretraining Foundation Model: Please see RSP;
100M-parameter Remote Sensing Unsupervised Pretraining Foundation Model: Please see RVSA;
Large-Scale RS Segmentation Pretraining Dataset: Please see SAMRS;
Other applications: ViTAE | VSA | QFormer | ViTPose | Matting | Scene Text Spotting | Video Object Segmentation
🔥 Update
2024.08.04
- 🏆 MTP got the highly cited paper!
2024.05.24
- Accepted by IEEE JSTARS Special issue on "Large-Scale Pretraining for Interpretation Promotion in Remote Sensing Domain"
2024.03.30
- The codes, configs and logs are released!
2024.03.29
- The change detection finetuned models are released!
2024.03.29
- The semantic segmentation finetuned models are released!
2024.03.28
- The rotated object detection finetuned models are released!
2024.03.28
- The horizontal object detection finetuned models are released!
2024.03.27
- The scene classification finetuned models are released!
2024.03.26
- The pretrained models are released!
2024.03.25
2024.03.21
- The paper is post on arxiv!
🌞 Overview
This is the official repository of the paper: MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining
In this study, we explore the Multi-Task Pretraining (MTP) paradigm for RS foundation models. Using a shared encoder and task-specific decoder architecture, we conduct multi-task supervised pretraining on the SAMRS dataset, encompassing semantic segmentation, instance segmentation, and rotated object detection. MTP supports both convolutional neural networks and vision transformer foundation models with over 300 million parameters. The pretrained models are finetuned on various RS downstream tasks, such as scene classification, horizontal and rotated object detection, semantic segmentation, and change detection. We hope this research encourages further exploration of RS foundation models and anticipate the widespread application of these models across diverse fields of RS image interpretation.
📖 Datasets and Models
Pretraining Dataset
We clip the DOTA-2.0 rotated bounding box version and produce the segmentation label by SAM, obtaining SOTA-RBB. (original SAMRS uses DOTA-2.0 horizontal bounding box version)
SOTA-RBB and the SIOR and FAST of original SAMRS is together used for implementing MTP.
We have uploaded SOTA-RBB to OneDive and Baidu.
Pretrained Models
| Pretrain | Pretraining Dataset | Backbone | Backbone Weights | Model Weights |
|---|---|---|---|---|
| MAE | Million-AID | ViT-L | Baidu & OneDrive | - |
| MAE + MTP | SAMRS | ViT-B+RVSA | Baidu & OneDrive | Baidu & OneDrive |
| MAE + MTP | SAMRS | ViT-L+RVSA | Baidu & OneDrive | Baidu & OneDrive |
| IMP + MTP | SAMRS | InternImage-XL | Baidu & OneDrive | Baidu & OneDrive |
Finetuned Models
Scene Classification
| Pretrain | Dataset | Backbone | OA | Config | Log | Weights |
|---|---|---|---|---|---|---|
| MAE + MTP | EuroSAT | ViT-B＋RVSA | 98.76 | Config | Log | Baidu & OneDrive |
| MAE + MTP | EuroSAT | ViT-L＋RVSA | 98.78 | Config | Log | Baidu & OneDrive |
| IMP + MTP | EuroSAT | InternImage-XL | 99.24 | Config | Log | Baidu & OneDrive |
| MAE + MTP | RESISC-45 | ViT-B＋RVSA | 95.57 | Config | Log | Baidu & OneDrive |
| MAE + MTP | RESISC-45 | ViT-L＋RVSA | 95.88 | Config | Log | Baidu & OneDrive |
| IMP + MTP | RESISC-45 | InternImage-XL | 96.27 | Config | Log | Baidu & OneDrive |
Horizontal Object Detection
| Pretrain | Dataset | Backbone | Method | AP50 | Config | Log | Weights |
|---|---|---|---|---|---|---|---|
| MAE + MTP | Xview | ViT-B＋RVSA | RetinaNet | 16.40 | Config | Log | Baidu & OneDrive |
| MAE + MTP | Xview | ViT-L＋RVSA | RetinaNet | 19.40 | Config | Log | Baidu & OneDrive |
| IMP + MTP | Xview | InternImage-XL | RetinaNet | 18.20 | Config | Log | Baidu & [OneDrive |