Latent Graph Representations for Surgical Scene Understanding

News

• [ 17/09/2025 ]: Check out our CAMMA Dataset Overlaps repository for an analysis of video overlaps across Cholec80, CholecT50, and Endoscapes to ensure fair dataset splits.

This repository contains the code corresponding to our Transactions on Medical Imaging paper Latent Graph Representations for Critical View of Safety Assessment, our MICCAI paper Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object- and Anatomy-Driven Reasoning, and our IPCAI/IJCARS paper Optimizing Latent Graph Representations for Unseen Domain Generalization.

[1] Latent Graph Representations for Critical View of Safety Assessment. Aditya Murali, Deepak Alapatt, Pietro Mascagni, Armine Vardazaryan, Alain Garcia, Nariaki Okamoto, Didier Mutter, Nicolas Padoy. IEEE Transactions on Medical Imaging 2023

[2] Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning. Aditya Murali, Deepak Alapatt, Pietro Mascagni, Armine Vardazaryan, Alain Garcia, Nariaki Okamoto, Didier Mutter, Nicolas Padoy. MICCAI 2023

[3] Optimizing Latent Graph Representations for Unseen Domain Generalization. Siddhant Satyanaik, Aditya Murali, Deepak Alapatt, Xin Wang, Pietro Mascagni, Nicolas Padoy. IJCARS 2024

News

In this repo we provide:

• Implementations of 3 different object detectors (Faster-RCNN, Cascade-RCNN, Deformable-DETR) and 3 different instance segmentation models (Mask-RCNN, Cascade-Mask-RCNN, Mask2Former) using the mmdetection framework.
• Implementations of 5 different object-centric models for fine-grained classification (e.g. CVS prediction). These comprise 4 methods introduced in [1]: LatentGraph-CVS (LG-CVS), DeepCVS, LayoutCVS, and ResNet50-DetInit, each of which can be run using any of the aforementioned object detection/segmentation models, as well as a 5th method, LatentGraph-DomainGen (LG-DG) introduced in [3].
• Implementation of a simple classifier using mmengine + mmdetection.
• Implementations of 2 different spatiotemporal object-centric models introduced in [2]: Surgical Videos as Latent SpatioTemporal Graphs (SV2LSTG), and DeepCVS-Temporal (DC-Temp).
• Annotation files and instructions to setup 3 different datasets: Endoscapes, Cholec80 (with CholecSeg8k segmentation masks), and CholecT50
• Config files and instructions to train/evaluate object detectors/segmentation models on Endoscapes2023 [4] and CholecSeg8k [5].
• Config files and instructions to train/evaluate the 5 single frame and 2 spatiotemporal methods on three tasks/datasets: CVS Prediction (Endoscapes), Phase Recognition (Cholec80), and Action Triplet Recognition (CholecT50)
• Trained model checkpoints for all tasks (coming soon).

Get Started

Installation

This project uses Pytorch 2.1.0 + CUDA 11.8, DGL 1.1.1, torch-scatter, mmdetection 3.2.0, and mmengine 0.7.4. Please note that you may encounter issues if you diverge from these versions. If you must diverge, please ensure that the DGL and torch-scatter versions match your versions of pytorch, and make sure to use mmengine<=0.7.4.

# clone mmdetection and export environment variable
> cd $HOME && git clone https://github.com/open-mmlab/mmdetection.git
> export MMDETECTION=$HOME/mmdetection

# clone SurgLatentGraph
> cd $HOME && git clone https://github.com/CAMMA-public/SurgLatentGraph.git
> cd SurgLatentGraph

# download pretrained weights
> cd weights
> wget -O coco_init_wts.zip https://seafile.unistra.fr/f/71eedc8ce9b44708ab01/?dl=1 && unzip coco_init_wts.zip && cd ..

# add surglatentgraph to PYTHONPATH to enable registry to find custom modules (note that this can be added to the .bashrc file for future use)
> export PYTHONPATH="$PYTHONPATH:$HOME/SurgLatentGraph"

# install dependencies 
> conda create -n latentgraph python=3.8 && conda activate latentgraph
(latentgraph) > conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
(latentgraph) > conda install -c dglteam/label/cu113 dgl
(latentgraph) > pip install torch-scatter -f https://data.pyg.org/whl/torch-2.0.1+cu117.html
(latentgraph) > pip install -U openmim
(latentgraph) > mim install mmdet
(latentgraph) > mim install mmengine==0.7.4
(latentgraph) > pip install torchmetrics
(latentgraph) > pip install scikit-learn
(latentgraph) > pip install prettytable
(latentgraph) > pip install imagesize
(latentgraph) > pip install networkx
(latentgraph) > pip install opencv-python
(latentgraph) > pip install yapf==0.40.1

Dataset Setup

Each dataset needs to be set up in the appropriate format. All of our models require frames to be extracted (at 1 fps) and use a modified COCO-style annotation structure. Each split of each dataset contains three JSON files: - annotation_coco.json is used to train object detectors, and includes only frames which have bounding box/segmentation ground truth. - annotation_ds_coco.json is used to train the single-frame downstream models, and includes all frames with labels for the downstream task. - annotation_coco_vid.json is used to train the spatiotemporal downstream models, and includes all frames at 1 fps from each dataset, regardless of whether they contain downstream labels.

All three files follow the normal COCO format, with three additional image level tags: - is_det_keyframe is a boolean value indicating whether the given frame contains ground-truth bounding box annotations. - is_ds_keyframe is a boolean value indicating whether the given frame contains a ground-truth downstream annotation. - ds contains the downstream annotation, which in our cases, can either be a list (CholecT50 triplet, Endoscapes CVS) for multilabel tasks or an integer (Cholec80 Phase) for single-frame tasks. In practice, for frames where is_ds_keyframe is False, we include the label from the last labeled frame in the video (ensures causality).

Dataset/Annotation Downloads

The Cholec80 and CholecT50 dataset download links contain entire surgical videos. To use them with this repository, the frames need to be extracted and named in the correct format, and our modified COCO-style annotations need to be downloaded. To guide this process, we provide example dataset folders with symbolic links in place of images, re-organized metadata for each dataset (all_metadata.csv), and the JSON-style annotations using the COCO Style Annotations link.

Example Setup

mkdir -p data/mmdet_datasets/
cd data/mmdet_datasets

# TODO: Download the annotations for each dataset using the COCO-style Annotations link.
unzip cholec80.zip && rm cholec80.zip
cd cholec80 && mkdir frames
# TODO: Extract frames at 25 fps and organize into the following directory structure
# - frames
#   - video01
#       - 0.jpg
#       - 1.jpg
#       ...
#   ...

# create symlinks
python copy_images.py

unzip cholecT50.zip && rm cholecT50.zip
cd cholecT50 && mkdir frames
# TODO: Extract frames at 25 fps and organize into the following directory structure
# - frames
#   - video01
#       - 0.jpg
#       - 1.jpg
#       ...
#   ...

# create symlinks
python copy_images.py

# For Endoscapes, the dataset is released in the same format we use in this repository, so you can just extract the files directly.
unzip endoscapes.zip && rm endoscapes.zip

The final directory structure should be as follows, with all symbolic links pointing to downloaded/extracted frames.

data/mmdet_datasets
└── endoscapes/
    └── train/
        └── 1_14050.jpg
        ...
        └── 120_40750.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── val/
        └── 121_23575.jpg
        ...
        └── 161_39400.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── test/
        └── 162_1225.jpg
        ...
        └── 201_55250.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── train_seg/
        └── 100_27200.jpg
        ...
        └── 98_65650.jpg
        └── annotation_coco.json
    └── val_seg/
        └── 126_11550.jpg
        ...
        └── 159_60800.jpg
        └── annotation_coco.json
    └── test_seg/
        └── 165_23650.jpg
        ...
        └── 189_34800.jpg
        └── annotation_coco.json
    └── train_vids.txt
    └── val_vids.txt
    └── test_vids.txt
    └── train_seg_vids.txt
    └── val_seg_vids.txt
    └── test_seg_vids.txt
└── cholec80/
    └── train_phase/
        └── 1_0.jpg
        └── 1_25.jpg
        └── 1_50.jpg
        ...
        └── 30_0.jpg
        ...
        └── 40_55525.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── val_phase/
        └── 41_0.jpg
        ...
        └── 48_45825.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── test_phase/
        └── 49_0.jpg
        ...
        └── 80_43075.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── train_vids.txt
    └── val_vids.txt
    └── test_vids.txt
└── cholecT50/
    └── train/
        └── 1_0.jpg
        ...
        └── 42_92775.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── val/
        └── 5_0.jpg
        ...
        └── 74_40825.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json
    └── test/
        └── 92_0.jpg
        ...
        └── 111_53625.jpg
        └── annotation_coco.json
        └── annotation_ds_coco.json
        └── annotation_coco_vid.json

Config Structure

Each dataset | detector | downstream_method combination has its own configuration. We summarize the config structure below. ```shell configs/ └── models/ └── endoscapes/ └── lg_base_box.py └── lg_base_seg.py └── lg_ds_base.py └── lg_save_base.py └── deepcvs_base.py └── simple_classifier.py └── simple_classifier_with_recon.py └── ssl └── simple_classifier_${INIT}.py ... # ResNet50 with different backbone initializations └── c80_phase/ └── lg_base_box.py ... # same files as endoscapes └── cholecT50/ └── lg_base_box.py ... # same files as endoscapes └── ${DETECTOR}/ # e.g. faster_rcnn └── lg_${DETECTOR}.py └── lg_ds_${DETECTOR}.py └── lg_ds_${DETECTOR}no_recon.py └── lg_save${DETECTOR}.py └── lg_ft_save_${DETECTOR}.py └── layout_${DETECTOR}.py └── layout_${DETECTOR}no_recon.py └── dc${DETECTOR}.py └── dc_${DETECTOR}no_recon.py ... # some ablations ... # one folder for each detector └── select_dataset.sh └── temporal_models/ └── endoscapes/ └── sv2lstg_model_base.py └── sv2lstg_5_base.py # sv2lstg └── sv2lstg_10_base.py └── sv2lstg_15_base.py └── sv2lstg_load_graphs_5_base.py # sv2lstg, skip image -> graph encoding and load saved graph └── sv2lstg_load_graphs_10_base.py └── sv2lstg_load_graphs_15_base.py └── dc_temp_model_base.py └── dc_temp_5_base.py # deepcvs-temporal └── dc_temp_10_base.py └── dc_temp_15_base.py └── cholecT50/ └── ... # same as endoscapes └── c80_phase/ └── ... # same as endoscapes └── sv2lstg_load_graphs_all.py # load all graphs in video -> temporal decoding to predict phase └── ${DETECTOR}/ # e.g. faster_rcnn └── sv2lstg${DETECTOR}5.py # use ${DETECTOR} to construct each latent graph, clips of 5 frames └── sv2lstg${DETECTOR}10.py └── sv2lstg${DETECTOR}15.py └── sv2lstg_lin_probe${DETECTOR}5.py # load latent graphs constructed with ${DETECTOR}, linear probing with clips of 5 frames └── sv2lstg_lin_probe${DETECTO