OpenVision Series

OpenVision (ICCV 2025)

A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

OpenVision 2 (CVPR 2026)

A Family of Generative Pretrained Visual Encoders for Multimodal Learning

OpenVision 3

A Unified Visual Encoder for Both Understanding and Generation

🌐 OpenVision Project Page
• Arxiv
• 💻 Code
• OpenVision Collection

🌐 OpenVision 2 Project Page
• Arxiv
• 💻 Code
• OpenVision 2 Collection

🌐 OpenVision 3 Project Page
• Arxiv
• 💻 Code
• OpenVision 3 Collection

This repository contains the code for training and fine-tuning vision-language models based on the OpenVision framework. It now supports both the original contrastive + generative training (OpenVision), the simplified caption-only generative training (OpenVision 2), providing efficient and scalable approaches to multimodal learning on TPU infrastructure.

🚀 Recent Updates

August 2026

• 📝 Released the OpenVision 2 caption text decoders: every *-vision-only repo now also ships the jointly-trained decoder, so encoder + decoder form the full image→caption model. See Generate Captions and caption.py.

January 2026

• ✨ Released OpenVision 3: a unified visual encoder for both understanding and generation.
• Please refer to the script for OpenVision 3 usage.
• The full training code will be released soon.

September 2025

• Released OpenVision 2: a simplified, generative-only version of OpenVision that removes the text encoder and contrastive loss, keeping only the captioning objective.
• OpenVision 2 achieves:
• 1.5–2× faster training
• ~1.8× lower memory footprint
• Supports scaling up to 1B+ parameters
• Maintains or improves performance on multimodal benchmarks (OCR, TextVQA, ChartQA, MME, etc.).

May 2025

• Released OpenVision models and training code.

🧩 OpenVision 2 at a Glance

• Architecture: Vision Encoder (ViT) + Text Decoder (no text encoder)
• Training Objective: Caption-only autoregressive generation
• Key Optimizations:
• Dual-stage CLIPA-style training (low → high resolution)
• Synthetic captions from ReCap-DataComp-1B v2 (LLaMA-3-powered, conditioned on alt-text)
• Visual token masking (keep ~25–35% tokens) for efficiency
• Efficiency:
• ViT-L/14 @224: Training time reduced from 83h → 57h, memory 24.5GB → 13.8GB
• SoViT-400M/14 @384: Training time 241h → 121h, memory 27.4GB → 14.5GB
• Enables larger batch size on TPU v4

📦 Core Features (Shared by OpenVision & OpenVision 2)

• Optimized for Google Cloud TPU training
• Supports various encoder architectures (ViT models of different sizes)
• Implements efficient training strategies including model sharding
• Supports pre-training and multi-stage fine-tuning
• Compatible with CLIP-style vision-language training

📖 OpenVision (Original)

• Training Objective: Contrastive (CLIP-style) + Generative (captioning)
• Highlights:
• Strong performance across multimodal benchmarks
• Public release of both code and pretrained weights
• Serves as the foundation for OpenVision 2

📊 Model Zoo (OpenVision 2)

OpenVision 2 Performance on Multimodal Benchmarks

Full collection: Hugging Face – OpenVision 2

🔧 How to Load Converted Vision Encoder

Note:
OpenVision2 checkpoints require the custom open_clip version included in this repository.
The upstream open_clip pip package is not compatible.

Example

import torch
# Use the OpenVision2 version of open_clip
from src.convert_upload.open_clip.factory import create_vision_encoder_and_transforms

hf_repo = "UCSC-VLAA/openvision2-vit-large-patch14-224-vision-only"

vision_encoder = create_vision_encoder_and_transforms(
    model_name=f"hf-hub:{hf_repo}"
)

vision_encoder.eval()
dummy_image = torch.ones((1, 3, 224, 224))
with torch.no_grad():
    _, patch_features = vision_encoder(dummy_image)

print("Patch feature shape:", patch_features.shape)

📝 Generate Captions (Full Generative Model)

Every OpenVision 2 *-vision-only repo now also ships the caption text decoder it was jointly trained with, so the encoder + decoder together form the full image→caption model. The decoder is a concat / prefix-LM autoregressive transformer (the image patch tokens are prepended as a bidirectional prefix; text is generated causally). Each repo contains caption_decoder.safetensors, text_decoder_config.json, and modeling_openvision2_decoder.py.

One-command demo (downloads encoder + decoder from the Hub; resolution is read automatically):

python caption.py --repo UCSC-VLAA/openvision2-vit-large-patch14-224-vision-only --image your.jpg

Or in code:

import json, torch, numpy as np
from PIL import Image
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from src.convert_upload.open_clip.factory import create_vision_encoder_and_transforms
from src.convert_upload.modeling_openvision2_decoder import (
    OpenVision2TextDecoder, OpenVision2TextDecoderConfig)

repo = "UCSC-VLAA/openvision2-vit-large-patch14-224-vision-only"
enc = create_vision_encoder_and_transforms(model_name=f"hf-hub:{repo}").eval()
d = json.load(open(hf_hub_download(repo, "text_decoder_config.json")))
dec = OpenVision2TextDecoder(OpenVision2TextDecoderConfig(
    width=d["width"], depth=d["depth"], num_heads=d["num_heads"], mlp_dim=d["mlp_dim"],
    vocab_size=d["vocab_size"], vision_width=d["vision_width"]))
dec.load_state_dict(load_file(hf_hub_download(repo, "caption_decoder.safetensors"))); dec.eval()

# image (ImageNet-normalized, NCHW) -> patch tokens -> caption ids
img = torch.randn(1, 3, 224, 224)  # replace with a preprocessed image; see caption.py
with torch.no_grad():
    _, patch_tokens = enc(img)
    ids = dec.generate(patch_tokens, bos_id=1, eos_id=2, max_len=64)[0].tolist()
# decode ids with assets/bert_base_vocab_bos_eos.txt (see caption.py: detokenize)

See caption.py for image preprocessing and WordPiece detokenization.

📊 Model Zoo (OpenVision)

Vision Encoder Performance on ImageNet-1K