github.com/ai4colonoscopy/IntelliScope @main

<img src="https://github.com/ai4colonoscopy/IntelliScope/raw/main/assets/project-name.jpeg"/>












    🔬 <b>Paper</b> (🔗 <a href="https://arxiv.org/abs/2410.17241">arXiv</a>, 🔗 <a href="https://link.springer.com/article/10.1007/s11633-025-1597-6">Springer</a>, 🤗 <a href="https://huggingface.co/papers/2410.17241">Huggingface</a>, 🤖 <a href="https://www.aimodels.fyi/papers/arxiv/frontiers-intelligent-colonoscopy">AIModels.fyi</a>  | 
    📖 <b>ColonSurvey</b> (🔗 <a href="https://docs.google.com/spreadsheets/d/1V_s99Jv9syzM6FPQAJVQqOFm5aqclmrYzNElY6BI18I/edit?usp=sharing">Online Sheet</a>) |
    🏥 <b>ColonINST</b> (🔗 <a href="https://drive.google.com/drive/folders/1ng2DQav-Gfts6hIr3_vCUC-a2gCWzzCO?usp=sharing">Google Drive</a>, 🤗 <a href="https://huggingface.co/datasets/ai4colonoscopy/ColonINST-v1">Huggingface</a>, <img src="https://github.com/ai4colonoscopy/IntelliScope/raw/main/assets/wisemodel.png" style="width:40px"> <a href="https://wisemodel.cn/datasets/Jingyi/ColonINST">wisemodel</a>) | 
    🤖 <b>ColonGPT</b> (🔗 <a href="https://drive.google.com/drive/folders/1Emi7o7DpN0zlCPIYqsCfNMr9LTPt3SCT?usp=drive_link">Google Drive</a>, 🤗 <a href="https://huggingface.co/ai4colonoscopy/ColonGPT">Huggingface</a>, <img src="https://github.com/ai4colonoscopy/IntelliScope/raw/main/assets/wisemodel.png" style="width:40px"> <a href="https://wisemodel.cn/models/Jingyi/ColonGPT">wisemodel</a>) |
    🏇 <b>Multimodal benchmark</b> (🔗 <a href="https://drive.google.com/drive/folders/1q3awr-aT50tuhW9Z01C3LKkckfG4Bk70?usp=sharing">Google Drive</a>, 🔗 <a href="https://paperswithcode.com/dataset/coloninst-v1">PaperWithCode</a>)






    <i>Keyworks: Intelligent Colonoscopy, Multimodal Colonoscopy Dataset, Multimodal Language Model, Vision-language Understanding, Endoscopic Image Analysis, Healthcare AI, Abdomen.</i>

Colonoscopy is currently one of the most sensitive screening methods for colorectal cancer (🔗 Wikipedia). Have you ever wondered how to make colonoscopy smarter? Well, buckle up, let's enter the exciting world of intelligent colonoscopy!

• Our vision. To explore the frontiers of intelligent colonoscopy techniques and their potential impact on multimodal medical applications.
• Why use IntelliScope? It combines "Intelligent" and "colonoScope", where "Intelli" reflects the intelligent processing and decision-making capabilities of the system, and "Scope" refers to the colonoscope device used in medical endoscopy. Together, they imply a cutting-edge multimodal system designed to improve colonoscopy with advanced AI technologies.
• Project members. Ge-Peng Ji (🇦🇺 ANU), Jingyi Liu (🇯🇵 Keio), Peng Xu (🇨🇳 THU), Nick Barnes (🇦🇺 ANU), Fahad Shahbaz Khan (🇦🇪 MBZUAI), Salman Khan (🇦🇪 MBZUAI), Deng-Ping Fan (🇨🇳 NKU)
• Let's join our IntelliScope community. We are building a discussion forum for the convenience of researchers to 💬 ask any questions, 💬 showcase/promote your work, 💬 access data resources, and 💬 share research ideas.

• Quick view. Next, we present some features of our colonoscopy-specific AI chatbot, ColonGPT. This is a domain-pioneering multimodal language model that can help endoscopists perform various user-driven tasks through interactive dialogues.

  

Updates

• [Mar/09/2026] We provide the translated version (CN) of the paper for the convenience of Chinese readers, enabling a broader audience to better understand the methodology, experimental results, and key contributions presented in this work.
• [Jan/07/2026] Our paper has officially available at Springer Nature, please read our paper with this link, and cite our paper using this bibtex.
• [Dec/09/2025] 🔥🔥🔥 Thrilled to announce the largest multimodal colonoscopy dataset, ColonVQA, with 1.1M+ VQA entries. We also propose the first reasoning-centric solutions: ColonReason dataset and the first reasoning-based model, ColonR1. Project is here: https://github.com/ai4colonoscopy/Colon-X. Your 🌟star is our biggest motivation to move forward.
• [April/05/2025] Our project now supports the Chinese AI platform wisemodel.
• [Feb/07/2025] We announce a new two-stage training strategy for enhancing ColonGPT's performance, achieving SOTA results on all downstream tasks. Further details are available in our technical report (arXiv-v2).
• [Oct/30/2024] We've set up an online benchmark on the paper-with-code website.
• [Oct/16/2024] Open-source the whole codebase of project.

🔥 Research Highlights

<img src="https://github.com/ai4colonoscopy/IntelliScope/raw/main/assets/overview_for_github.png"  width="800px" />


<em> 
Figure 1: Introductary diagram.
</em>

• Survey on colonoscopic scene perception (CSP) ➡️ "We assess the current landscape to sort out domain challenges and under-researched areas in the AI era."
• 📖 ColonSurvey. We investigate the latest research progress in four colonoscopic scene perception tasks from both data-centric and model-centric perspectives. Our investigation summarises key features of 63 datasets and 137 representative deep techniques published since 2015. In addition, we highlight emerging trends and opportunities for future study. (🔗 Hyperlink)
• 💥 Multimodal AI Initiatives ➡️ "We advocate three initiatives to embrace the coming multimodal era in colonoscopy."
• 🏥 ColonINST. We introduce a pioneering instruction tuning dataset for multimodal colonoscopy research, aimed at instructing models to execute user-driven tasks interactively. This dataset comprises of 62 categories, 300K+ colonoscopic images, 128K+ medical captions (GPT-4V) generated), and 450K+ human-machine dialogues. (🔗 Hyperlink)
• 🤖 ColonGPT. We develop a domain-specific multimodal language model to assist endoscopists through interactive dialogues. To ensure reproducibility for average community users, we implement ColonGPT in a resource-friendly way, including three core designs: a 0.4B-parameter visual encoder 🤗 SigLIP-SO from Google, a 1.3B-parameter lightweight language model 🤗 Phi1.5 from Microsoft, and a multigranularity adapter for token reducing from 100% to only 33.74% but not compromise to performance. (🔗 Hyperlink)
• 💯 Multimodal Benchmark. We contribute a multimodal benchmark, including six general-purpose models and two designed for medical purposes, across three colonoscopy tasks to enable fair and rapid comparisons going forward. (🔗 Hyperlink)

📖 ColonSurvey

Our "ColonSurvey" project contributes various useful resources for the community. We investigate 63 colonoscopy datasets and 137 deep learning models focused on colonoscopic scene perception, all sourced from leading conferences or journals since 2015. This is a quick overview of our investigation; for a more detailed discussion, please refer to our paper in PDF format.

<img src="https://github.com/ai4colonoscopy/IntelliScope/raw/main/assets/colonsurvey.png"/>


<em> 
Figure 2: The investigation of colonoscopy datasets and models.
</em>

To better understand developments in this rapidly changing field and accelerate researchers’ progress, we are building a 📖 paper reading list, which includes a number of AI-based scientific studies on colonoscopy imaging from the past 12 years. [UPDATE ON OCT-14-2024] In detail, our online list contains:

• Colonoscopy datasets 🔗 Google sheet
• Colonoscopy models
• Classification tasks 🔗 Google sheet
• Detection tasks 🔗 Google sheet
• Segmentation tasks 🔗 Google sheet
• Vision language tasks 🔗 Google sheet
• 3D analysis tasks (*supplementary content) 🔗 Google sheet

Make our community great again. If we miss your valuable work in google sheet, please add it and this project would be a nice platform to promote your work. Or anyone can inform us via email (📮 gepengai.ji@gmail.com) or push a PR in github. We will work on your request as soon as possible. Thank you for your active feedback.

🏥 ColonINST (A multimodal instruction tuning dataset)

<img src="https://github.com/ai4colonoscopy/IntelliScope/raw/main/assets/coloninst-overview.png"/>


<em> 
Figure 3: Details of our multimodal instruction tuning dataset, ColonINST. (a) Three sequential steps to create the instruction tuning dataset for multimodal research. (b) Numbers of colonoscopy images designated for training, validation, and testing purposes. (c) Data taxonomy of three-level categories. (d) A word cloud of the category distribution by name size. (e) Caption generation pipeline using the VL prompting mode of GPT-4V. (f) Numbers of human-machine dialogues created for four downstream tasks.
</em>

Our data contains two parts: colonoscopy images and human-machine dialogues (available at 🤗 huggingface and 🔗 google drive). However, due to privacy-preserving concerns, we can not directly share the origin medical images without its authorization. DO NOT WORRY! We prepare a data download list and an easy-to-use script to organise our ColonINST. The operation instructions are detailed in our document (🔗 ./docs/guideline-for-ColonINST.md)

Apply full data of the proposed ColonINST via google form: 🈸 https://forms.gle/C3FqtnCZmo5aZLR26

🤖 ColonGPT (A colonoscopy-specific multimodal Language Model)

<img src="https://github.com/ai4colonoscopy/IntelliScope/raw/main/assets/ColonGPT.gif" width="666px"/>


<em> 
Figure 4: Details of our multimodal language model, ColonGPT.
</em>

Our ColonGPT is a standard multimodal language model, which has been released at 🔗 google drive platform. It contains four basic components: a language tokenizer, an visual encoder (🤗 SigLIP-SO or 🔗 google drive), a multimodal connector, and a language model (🤗 Phi1.5 or 🔗 google drive).

✅ Quick start

We show a code snippet to show you how to quickly try-on our ColonGPT model with HuggingFace transformers quickly. For convenience, we manually combined some configuration and code files and merged the weights. Please note that this is a quick code, we recommend you installing ColonGPT's source code to explo