🎥 Tempo: Small Vision-Language Models are Smart Compressors for Long Video Understanding, ECCV 2026

Tempo is an efficient, query-aware framework that natively compresses hour-long videos for downstream Multimodal LLMs. Instead of blindly dropping frames, Tempo acts as an intelligent temporal compressor, dynamically distributing the rhythm of the video based on user intent.

Project Page | Paper | Demo

✨ Highlights

• 🧠 Intent-Driven Compression (ATA): Uses a Small VLM as an O(1) dynamic router to allocate dense token bandwidth to query-critical moments while rapidly fast-forwarding redundant backgrounds.
• ⚡ Extreme Efficiency: Achieves aggressive dynamic compression (0.5–16 tokens/frame), bypassing the lost-in-the-middle phenomenon without breaking causality.
• 🏆 State-of-the-Art Performance: Our compact Tempo-6B model scores 52.3 on the extreme-long LVBench under a strict 8K visual token budget (53.7 with 12K budget), outperforming proprietary baselines like GPT-4o and Gemini 1.5 Pro.

---

🎬 Watch Tempo in Action

(Click play to see our interactive UI, dynamic token allocation visualization, and real-time inference)

---

🔥 News

• [2026.04] 📦 We have released the Intermediate Checkpoints for Stages 0, 1, and 2! You can find them in our Hugging Face Collection.
• [2026.04] 📊 The full Evaluation Pipeline is now open-sourced! Our evaluation scripts, integrated with the standard lmms-eval framework for LVBench, Video-MME, MLVU, and LongVideoBench, are ready to use. Please refer to the Evaluation Section for detailed instructions.
• [2026.04] 📄 Our paper is officially out! You can read it on arXiv and check out our page on Hugging Face Papers.
• [2026.04] 🚀 We have released the Tempo-6B inference code, interactive Gradio UI, and the final checkpoints (Stage 3)!
• [TODO] 🛠️ Training Code: The complete training scripts for all 4 stages will be open-sourced in the following weeks. Stay tuned!

⭐ Tip: Please Watch or Star this repository to keep an eye on our latest updates and code releases!

---

🚀 Quick Start

1. Installation

Create a new conda environment and install all required dependencies:

# Clone our repository
git clone https://github.com/FeiElysia/Tempo.git
cd Tempo

# Create environment
conda create -n tempo python=3.12 -y
conda activate tempo

# Install all packages (PyTorch 2.6.0 + CUDA 12.4)
pip install -r requirements.txt

⚡ Installing Flash-Attention

Since flash-attn installation can be highly environment-dependent, please install it manually using one of the methods below:

# Method 1
pip install flash-attn==2.7.4.post1

# Method 2: Without Build Isolation
pip install flash-attn==2.7.4.post1 --no-build-isolation

# Method 3: If you are unable to build from source, you can directly download and install the pre-built wheel:
wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl
pip install flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl
rm flash_attn*.whl

💡 If you are unable to install flash-attn, you can still run Tempo by disabling it: 1. Set use_flash_attn=False when calling load_pretrained_model. 2. Comment out the line self.config._attn_implementation = "flash_attention_2" in qwen3vl_encoder.py.

Theoretically, the numerical differences should be minimal. We have visually verified that the model produces excellent qualitative results without Flash-Attention. However, please note that we have not rigorously evaluated its impact on the benchmarks we reported in the paper.

2. Model Zoo

To fully support the open-source community and facilitate future research, we have released the weights for our final model alongside the intermediate checkpoints from all 4 stages of our training pipeline.

💡 Note on Token Budgets: Tempo's Adaptive Token Allocation (ATA) is dynamically controlled at inference time. The 4K and 8K budget configurations reported in our paper use the exact same final weights (Stage 3). You simply adjust the budget hyperparameter during inference.

Training Stage	Description	Weights
Stage 0	Modality Alignment	🤗 HF Link
Stage 1	Pre-training	🤗 HF Link
Stage 2	Broad Supervised Fine-Tuning	🤗 HF Link
Stage 3	Long-Context SFT (Final Tempo-6B)	🤗 HF Link

(Note: If you only want to run inference or evaluate our model, simply download the Stage 3 weights. The intermediate checkpoints for Stages 0, 1, and 2 are provided for researchers who wish to reproduce our training pipeline, conduct ablation studies, or perform custom fine-tuning.)

3. Prepare Checkpoints

To run the inference script successfully, you need to download two components: our final Tempo-6B weights, and the base Qwen3-VL-2B-Instruct model (for Tempo initialization).

We highly recommend using the huggingface-cli for fast and resumable downloads:

mkdir -p checkpoints

# 1. Download the final Tempo-6B model
huggingface-cli download --resume-download Vision-CAIR/Tempo-6B --local-dir ./checkpoints/Tempo-6B

# 2. Download the base Qwen3-VL model (Required for architecture initialization)
# 💡 Note: To avoid caching Qwen3-VL in the default system drive during inference, 
# you can modify Tempo-6B's `config.json`: change "Qwen/Qwen3-VL-2B-Instruct" to "./checkpoints/Qwen3-VL-2B-Instruct" and run:
huggingface-cli download --resume-download Qwen/Qwen3-VL-2B-Instruct --local-dir ./checkpoints/Qwen3-VL-2B-Instruct

---

💻 Inference & Demos

We provide multiple ways to interact with Tempo, from web UI to batch scripts.

1. Web UI (Gradio)

To launch the local Gradio application with interactive visualizations of the Token Allocation distribution:

python app.py

Navigate to the generated local or public URL in your browser. Our UI features dynamic token compression visualization and one-click example testing.

2. Single Video Inference

Run the default example: We provide a quick-start script to test a pre-configured.

sh ./scripts/infer/infer.sh

Run your own custom video: To test your own videos, call the Python script directly. Make sure to point the --model_path to your downloaded local checkpoint.

python infer.py \
    --model_path "./checkpoints/Tempo-6B" \
    --video_path "/path/to/your/custom_video.mp4" \
    --query "Your detailed question here."

3. Batch Inference

Run all default examples: To sequentially reproduce all the qualitative examples shown on our Project Page, run:

sh ./scripts/infer/infer_all_demos.sh

⚠️ Note: For long demo videos (e.g., lvbench_gXnhqF0TqqI.mp4 and videomme_Sp2nxlrQ89w.mp4), you have two options to avoid "file not found" errors: * Option 1 (Download): Download them manually from our Examples Release Page and place them in the examples/ directory. * Option 2 (Skip): Simply open examples/demo_cases.json and delete their corresponding entries.

Run a custom batch: For testing across multiple custom videos, we highly recommend our JSON-based pipeline. Simply edit the test configurations in ./examples/demo_cases.json:

[
  {
    "video_path": "/path/to/custom1.mp4",
    "query": "Question 1"
  },
  {
    "video_path": "/path/to/custom2.mp4",
    "query": "Question 2"
  }
]

Once configured, execute the same batch script to automatically process them all:

sh ./scripts/infer/infer_all_demos.sh

---

✨ See Tempo in Action

Here are some unedited inference results generated directly by our Tempo-6B model. These examples demonstrate the model's capability in video summarization and complex QA.

🧠 Example 1: Complex Multi-Task Analysis & Grounding (Honkai: Star Rail)

🔷 Query:

Task: Please analyze the provided video and answer the following 7 questions precisely. Q1: How many performers are visible on the stage? Q2: Describe the architectural elements in the background. What historical civilization do they remind you of? Q3: What is happening in the night sky above the performers, and what does this suggest about the event? Q4: List the hair colors of the performers in order from left to right. Q5: Identify the specific musical instrument being played by the performer located on the far left of the stage. Q6: What is the specific time interval (in seconds, e.g., XX-XXs) during which this fireworks performance scene occurs in the video? Q7: Look at the audience in the foreground. How does their silhouette-like depiction affect the viewer's perspective of the stage?

🔴 Tempo Response: ⏱️ Refer to 1:27 - 1:30

Q1: There are four performers visible on the stage.

Q2: The architectural elements in the background resemble ancient Greek columns, suggesting a historical setting reminiscent of ancient Greece.

Q3: The night sky above the performers is filled with fireworks, suggesting a celebratory event or performance.

Q4: The hair colors of the performers from left to right are purple, pink, blonde, and blue.

Q5: The performer on the far left is playing a harp.

Q6: The fireworks performance scene occurs between 88.0-91.0 seconds.

Q7: The silhouette-like depiction of the audience in the foreground creates a sense of depth and focus on the performers on stage, enhancing the viewer's perspective of the stage.

🔥 Example 2: Video Summarization (Honkai Impact 3rd)

🔷 Query: Describe the video in detail.

🔴 Tempo Response: ⏱️ Global Summary

The video opens with a high-tech, neon-lit cityscape at night, filled with floating structures and digital screens. A character with pink hair and a blue eye patch is seen running through the city, dodging obstacles. The environment is dark with vibrant neon colors, primarily purples, blues, and greens. The character is dressed in a futuristic outfit with a weapon. As the character moves, they are surrounded by digital elements and symbols, suggesting a cyberpunk setting. The scene shifts to a close-up of the character's face, showing determination and focus. The character is then seen in a combat stance, ready to fight. The lighting is dynamic, with flashes of light and energy. The video ends with a black screen displaying credits in white text.

---

🧠 Methodology: The Tempo Framework

Tempo bridges the gap between hour-long videos and bounded LLM context windows by casting long video understanding as an end-to-end, query-aware cross-modal distillation process.

Instead of blindly sampling frames, our pipeline operates in three highly efficient phases:

• 1. Local Compressor (Early Distillation): A Small VLM (SVLM) processes video segments alongside the user query. It uses learnable memory tokens to inherently distill visual semantics, dropping query-irrelevant backgrounds early on.
• 2. Adaptive Token Allocation (ATA): Operating as a training-free, O(1) dynamic router during inference, ATA intercepts zero-shot relevance scores from the SVLM. It allocates dense token bandwidth to query-critical segments while rapidly fast-forwarding redundancies into minimal temporal anchors.
• 3. Global Decoder (Synthesis): The highly compressed, filtered memory tokens are assembled with explicit temporal tags (e.g., <t=2.0s>). A large global LLM then synthesizes this condensed storyline to generate precise answers without suffering from attention dilution.

---

📊 Quantitative Results

Tempo achieves state-of-the-art performance on long video benchmarks while using a fraction of the token budget compared to traditional models.

Model	Size	Tokens / Frame	LongVideoBench	MLVU	Video-MME	LVBench
Proprietary Models
GPT-4o	-	-	66.7	64.6	71.9	30