DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding

The World's Top-Performing Vision Model for Open-World Object Detection

The project provides examples for using DINO-X, which is hosted on DeepDataSpace.

IDEA Research

Highlights

Beyond Grounding DINO 1.5, DINO-X has several improvements, taking a step forward towards becoming a more general object-centric vision model. The highlights of the DINO-X are as follows:

✨ The Strongest Open-Set Detection Performance: DINO-X Pro set new SOTA results on zero-shot transfer detection benchmarks: 56.0 AP on COCO, 59.8 AP on LVIS-minival and 52.4 AP on LVIS-val. Notably, it scores 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val benchmarks, improving the previous SOTA performance by 5.8 box AP and 5.0 box AP. Such a result underscores its significantly enhanced capacity for recognizing long-tailed objects.

🔥 Diverse Input Prompt and Multi-level Output Semantic Representations: DINO-X can accept text prompts, visual prompts, and customized prompts as input, and it outputs representations at various semantic levels, including bounding boxes, segmentation masks, pose keypoints, and object captions, with multiple perception heads.

🍉 Rich and Practical Capabilities: DINO-X can simultaneously support lots of highly practical tasks, including Open-Set Object Detection and Segmentation, Phrase Grounding, Visual-Prompt Counting, Pose Estimation, and Region Captioning. We further develop a universal object prompt to achieve Prompt-Free Anything Detection and Recognition.

🔌 Seamless AI Tool Integration: With DINO-X MCP Server, developers can integrate DINO-X's capabilities directly into Cursor, Claude, and other MCP-compatible AI tools, enabling object detection in conversational AI workflows.

Latest News

• 2025.07.23: We've updated dds-cloudapi-sdk to version 0.5.3, which significantly improves mask encoding by removing the previous non-standard method and adopting the pycocotools-aligned rle mask format. This change makes it much easier to decode masks directly with pycocotools, and we've added a new mask_format = coco_rle parameter to the API; you can find detailed usage examples here: dds visualization utils

• 2025.06.18: 🚀 DINO-X MCP Server is now available! Integrate DINO-X into Cursor and other MCP-compatible tools. Check dinox-mcp for details.

• 2025.05.21: For more demo usages, including DINO-X, T-Rex, DINO-X-SeeK, please check dds-cloud-api examples for more details.

• 2025.04.21: Update to dds-cloudapi-sdk API V2 version. The V1 version in the original API for DINO-X has been deprecated, please update to the latest dds-cloudapi-sdk by pip install dds-cloudapi-sdk -U to use DINO-X model. Please refer to dds-cloudapi-sdk and our API docs to view more details about the update.

• 2025.03.11: We have released DINO-XSeeK model towards detecting objects based on more complex user descriptions. Please refer to RexSeeK for more details and the demo has already been available at here.

• 2025.01.18: DINO-X achieves SOTA performance of 51.7 average mask AP score on Segmentation in the Wild zero-shot track.

• 2024.12.05: Released the Prompt-Free Anything Detection and Segmentation feature. For API usage and demo visualization, please refer to here. To use the latest features, please install dds-cloudapi-sdk==0.3.3.

• 2024.12.04: Launched the Open-World Detection and Segmentation feature. For API usage and demo visualization, visit here.

• 2024.12.03: Support DINO-X with SAM 2 for Open-World Anything Segmentation and Tracking. For more details, check out the Grounded SAM 2 project.

Contents

• Model Framework
• Model Performance
• Side-by-side Performance Comparison with Previous Best Methods
• Zero-Shot Performance on Object Detection Benchmarks
• Zero-Shot Performance on Generic Segmentation Benchmarks
• API Usage
• Installation
• Registration
• Support Demos
  • Open-World Detection and Segmentation
  • Prompt-Free Anything Detection and Segmentation
• Related Works
• BibTeX

Model Framework

DINO-X can accept text prompts, visual prompts, and customized prompts as input, and it can generate representations at various semantic levels, including bounding boxes, segmentation masks, pose keypoints, and object captions.

Performance

Side-by-Side Performance Comparison with Previous Best Methods

Zero-Shot Performance on Object Detection Benchmarks

Model	COCO (AP box)	LVIS-minival (AP all)	LVIS-minival (AP rare)	LVIS-val (AP all)	LVIS-val (AP rare)
Other Best Open-Set Model	53.4 (OmDet-Turbo)	47.6 (T-Rex2 visual)	45.4 (T-Rex2 visual)	45.3 (T-Rex2 visual)	43.8 (T-Rex2 visual)
DetCLIPv3	-	48.8	49.9	41.4	41.4
Grounding DINO	52.5	27.4	18.1	-	-
T-Rex2 (text)	52.2	54.9	49.2	45.8	42.7
Grounding DINO 1.5 Pro	54.3	55.7	56.1	47.6	44.6
Grounding DINO 1.6 Pro	55.4	57.7	57.5	51.1	51.5
DINO-X Pro	56.0	59.7	63.3	52.4	56.5

• Performance: DINO-X Pro achieves SOTA performance on COCO, LVIS-minival, LVIS-val, zero-shot object detection benchmarks.
• Effective Long-tail Object Detection: DINO-X Pro has significantly improved the model's performance on LVIS-rare classes, significantly surpassing the previous SOTA Grounding DINO 1.6 Pro model by 5.8 AP and 5.0 AP, respectively, demonstrating the exceptional capability of DINO-X in long-tailed object detection scenarios.

Zero-Shot Performance on Generic Segmentation Benchmarks

Model	COCO (AP mask)	LVIS-minival (AP mask)	LVIS-minival (AP mask rare)	LVIS-val (AP mask)	LVIS-val (AP mask rare)	SGinW (AP mask avg)
Assembled General Perception Model
Grounded HQ-SAM (Base + Huge)	-	-	-	-	-	49.6
Grounded SAM (1.5 Pro + Huge)	44.3	47.7	50.2	41.8	46.0	-
Grounded SAM 2 (1.5 Pro + Large)	44.7	46.2	50.1	40.5	44.6	-
DINO-X Pro + SAM-Huge	44.2	51.2	52.2	-	-	-
Unified Vision Model
DINO-X Pro (Mask Head)	37.9	43.8	46.7	38.5	44.4	51.7

• Performance: DINO-X achieves SOTA performance of 51.7 average mask AP on SGinW zero-shot benchmarks. And DINO-X also achieves mask AP scores of 37.9, 43.8, and 38.5 on the COCO, LVIS-minival, and LVIS-val zero-shot instance segmentation benchmarks, respectively.Compared to Grounded SAM and Grounded SAM 2, there is still a notable performance gap for DINO-X to catch up. We will further optimize the segmentation performance in the future release.
• Efficiency: Unlike Grounded SAM series, DINO-X significantly improves the segmentation efficiency by generating corresponding masks for each region without requiring multiple complex inference steps.
• Practical Usage: Users can use the mask function of DINO-X based on their actual needs. If the users require simultaneously object segmentation and tracking, we recommend using the latest Grounded SAM 2 (DINO-X + SAM 2), which we have already implemented in here.

API Usage

Installation

• Install the required packages

pip install -r requirements.txt

Note: If you encounter some errors with API, please install the latest version of dds-cloudapi-sdk:

pip install dds-cloudapi-sdk --upgrade

Register on Offical Website to Get API Token

• First-Time Application: If you are interested in our project and wish to try our algorithm, you will need to apply for the corresponding API Token through our request API token website for your first attempt.

• Request Additional Token Quotas: At this stage, we now support WeChat Pay as a payment channel. Users can purchase additional API calls through our official platform. If you encounter any issues during the purchase process or have other collaboration needs, feel free to contact us via this email address: deepdataspace_dm@idea.edu.cn.

Run local API demos

Open-World Object Detection and Segmentation

Open-world detection means users can detect anything with text prompts, try this feature by setting your API token in demo.py and run local demo:

python demo.py

After running the local demo, the annotated image will be saved at: ./outputs/open_world_detection

Demo Image Visualization

With the text prompt "wheel . eye . helmet . mouse . mouth . vehicle . steering wheel . ear . nose", we will get the predicton results as follows:

Demo Image	Box Prediction	Mask Prediction

Prompt-Free Anything Detection and Segmentation

We've implemented a novel Prompt Free object detection feature, which means users