OpenVision Series

OpenVision (ICCV 2025)

A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

OpenVision 2 (CVPR 2026)

A Family of Generative Pretrained Visual Encoders for Multimodal Learning

OpenVision 3

A Unified Visual Encoder for Both Understanding and Generation

🌐 OpenVision Project Page
• Arxiv
• 💻 Code
• OpenVision Collection

🌐 OpenVision 2 Project Page
• Arxiv
• 💻 Code
• OpenVision 2 Collection

🌐 OpenVision 3 Project Page
• Arxiv
• 💻 Code
• OpenVision 3 Collection

This repository contains the code for training and fine-tuning vision-language models based on the OpenVision framework. It now supports both the original contrastive + generative training (OpenVision), the simplified caption-only generative training (OpenVision 2), providing efficient and scalable approaches to multimodal learning on TPU infrastructure.

🚀 Recent Updates

January 2026

• ✨ Released OpenVision 3: a unified visual encoder for both understanding and generation.
• Please refer to the script for OpenVision 3 usage.
• The full training code will be released soon.

September 2025

• Released OpenVision 2: a simplified, generative-only version of OpenVision that removes the text encoder and contrastive loss, keeping only the captioning objective.
• OpenVision 2 achieves:
• 1.5–2× faster training
• ~1.8× lower memory footprint
• Supports scaling up to 1B+ parameters
• Maintains or improves performance on multimodal benchmarks (OCR, TextVQA, ChartQA, MME, etc.).

May 2025

• Released OpenVision models and training code.

🧩 OpenVision 2 at a Glance

• Architecture: Vision Encoder (ViT) + Text Decoder (no text encoder)
• Training Objective: Caption-only autoregressive generation
• Key Optimizations:
• Dual-stage CLIPA-style training (low → high resolution)
• Synthetic captions from ReCap-DataComp-1B v2 (LLaMA-3-powered, conditioned on alt-text)
• Visual token masking (keep ~25–35% tokens) for efficiency
• Efficiency:
• ViT-L/14 @224: Training time reduced from 83h → 57h, memory 24.5GB → 13.8GB
• SoViT-400M/14 @384: Training time 241h → 121h, memory 27.4GB → 14.5GB
• Enables larger batch size on TPU v4

📦 Core Features (Shared by OpenVision & OpenVision 2)

• Optimized for Google Cloud TPU training
• Supports various encoder architectures (ViT models of different sizes)
• Implements efficient training strategies including model sharding
• Supports pre-training and multi-stage fine-tuning
• Compatible with CLIP-style vision-language training

📖 OpenVision (Original)

• Training Objective: Contrastive (CLIP-style) + Generative (captioning)
• Highlights:
• Strong performance across multimodal benchmarks
• Public release of both code and pretrained weights
• Serves as the foundation for OpenVision 2

📊 Model Zoo (OpenVision 2)

OpenVision 2 Performance on Multimodal Benchmarks

Full collection: Hugging Face – OpenVision 2

🔧 How to Load Converted Vision Encoder

Note:
OpenVision2 checkpoints require the custom open_clip version included in this repository.
The upstream open_clip pip package is not compatible.

Example

import torch
# Use the OpenVision2 version of open_clip
from src.convert_upload.open_clip.factory import create_vision_encoder_and_transforms

hf_repo = "UCSC-VLAA/openvision2-vit-large-patch14-224-vision-only"

vision_encoder = create_vision_encoder_and_transforms(
    model_name=f"hf-hub:{hf_repo}"
)

vision_encoder.eval()
dummy_image = torch.ones((1, 3, 224, 224))
with torch.no_grad():
    _, patch_features = vision_encoder(dummy_image)

print("Patch feature shape:", patch_features.shape)

📊 Model Zoo (OpenVision)

Vision Encoder Performance on ImageNet-1K