This is the offical dataset for paper "RSVG: Exploring Data and Model for Visual Grounding on Remote Sensing Data", Paper.
School of Artificial Intelligence, OPtics, and ElectroNics (iOPEN), Northwestern Polytechnical University
Release the DIOR_RSVG dataset.
[2022/10/22]: Publish the manuscript on arXiv.
This is Multi-Granularity Visual Language Fusion (MGVLF) Network, the PyTorch source code of the paper "RSVG: Exploring Data and Model for Visual Grounding on Remote Sensing Data". It is built on top of the TransVG in PyTorch. Our method is a transformer-based method for visual grounding for remote sensing data (RSVG). It has achieved the SOTA performance in the RSVG task on our constructed RSVG dataset.
<img src="https://github.com/ZhanYang-nwpu/RSVG-pytorch/raw/main/fig/DIOR-RSVG.jpg">
| Dataset | train | val | test | Overall |
|---|---|---|---|---|
| Flickr30k [Paper] [Code] [Website] | 29783 (94%) | 1000 (3%) | 1000 (3%) | 31783 |
| ReferItGame [Paper] [Website] | 54127 (45%) | 5842 (5%) | 60103 (50%) | 120072 |
| RefCOCO [Paper][Code] | 120624 (85%) | 10834 (7%) | 5657 (3%) | 142210 |
| RefCOCO+ [Paper][Code] | 120191 (85%) | 10758 (7%) | 5726 (4%) | 141564 |
| GuessWhat [Paper] [Code] [Website] | 70% | 15% | 15% | 100% |
| Cops-Ref [Paper] [Code] | 119603 (80.5%) | 16524 (11%) | 12586 (8.5%) | 148713 |
| KB-Ref [Paper] [Code] | 31284 (72%) | 4000 (10%) | 8000 (18%) | 43284 |
| Ref-Reasoning [Paper] [Code] [Website] | 721164 (91%) | 36183 (4.6%) | 34609 (4.4%) | 791956 |
| RSVG [Paper] [Website] | 5505 (70%) | 1201 (15%) | 1227 (15%) | 7933 |
| DIOR-RSVG [Paper] [Dataset] | 26991 (70%) | 3829 (10%) | 7500 (20%) | 38320 |
<img src="https://github.com/ZhanYang-nwpu/RSVG-pytorch/raw/main/fig/MGVLF.jpg">
We recommended the following dependencies. - Python 3.6.13 - PyTorch 1.9.0 - NumPy 1.19.2 - cuda 11.1 - opencv 4.5.5 - torchvision
Download our constructed RSVG dataset files. We build the first large-scale dataset for RSVG, termed DIOR-RSVG, which can be downloaded from our Google Drive. The download link is available below:
https://drive.google.com/drive/folders/1hTqtYsC6B-m4ED2ewx5oKuYZV13EoJp_?usp=sharing
We expect the directory and file structure to be the following:
./ # current (project) directory
├── data_loader.py # Load data
├── main.py # Main code for training, validation, and test
├── README.md
└── DIOR_RSVG/ # DIOR-RSVG dataset
├── Annotations/ # Query expressions and bounding boxes
│ ├── 00001.xml/
│ └── ..some xml files..
├── JPEGImages/ # Remote sensing images
│ ├── 00001.jpg/
│ └── ..some jpg files..
├── train.txt # ID of training set （26991）
├── val.txt # ID of validation set （3829）
└── test.txt # ID of test set （7500）
Note that the dataset is ONLY permitted to be used for research.
If you found this code useful, please cite the paper. Welcome :+1:Fork and Star:+1:, then I will let you know when we update.
@ARTICLE{10056343,
author={Zhan, Yang and Xiong, Zhitong and Yuan, Yuan},
journal={IEEE Transactions on Geoscience and Remote Sensing},
title={RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data},
year={2023},
volume={61},
number={},
pages={1-13},
doi={10.1109/TGRS.2023.3250471}
}
Our DIOR-RSVG is constructed based on the DIOR remote sensing image dataset. We thank to the authors for releasing the dataset. Part of our code is borrowed from TransVG. We thank to the authors for releasing codes. I would like to thank Xiong zhitong and Yuan yuan for helping the manuscript. I also thank the School of Artificial Intelligence, OPtics, and ElectroNics (iOPEN), Northwestern Polytechnical University for supporting this work.
Licensed under a Creative Commons Attribution-NonCommercial 4.0 International for Non-commercial use only. Any commercial use should get formal permission first.
$ claude mcp add RSVG-pytorch \
-- python -m otcore.mcp_server <graph>