VTP (Visual Tokenizer Pre-training) Model. A unified framework supporting multiple vision tasks through composable methods: Basic feature extraction: - get_last_layer_feature(): Raw features from last layer - get_intermediate_layers_feature(): Multi-layer features CLIP
| 49 | |
| 50 | |
| 51 | class VTPModel(VTPPreTrainedModel): |
| 52 | """VTP (Visual Tokenizer Pre-training) Model. |
| 53 | |
| 54 | A unified framework supporting multiple vision tasks through composable methods: |
| 55 | |
| 56 | Basic feature extraction: |
| 57 | - get_last_layer_feature(): Raw features from last layer |
| 58 | - get_intermediate_layers_feature(): Multi-layer features |
| 59 | |
| 60 | CLIP zero-shot: |
| 61 | - get_clip_image_feature(): CLIP-projected image features |
| 62 | - get_clip_text_feature(): CLIP-projected text features |
| 63 | |
| 64 | Reconstruction: |
| 65 | - get_reconstruction_latents(): Bottleneck latents for reconstruction |
| 66 | - get_latents_decoded_images(): Decode latents to images |
| 67 | |
| 68 | Example: |
| 69 | >>> config = VTPConfig(vision_embed_dim=768, train_clip=True) |
| 70 | >>> model = VTPModel(config) |
| 71 | >>> |
| 72 | >>> # CLIP zero-shot |
| 73 | >>> img_feat = model.get_clip_image_feature(images) |
| 74 | >>> txt_feat = model.get_clip_text_feature(tokens) |
| 75 | >>> |
| 76 | >>> # Reconstruction |
| 77 | >>> latents = model.get_reconstruction_latents(images) |
| 78 | >>> reconstructed = model.get_latents_decoded_images(latents) |
| 79 | >>> |
| 80 | """ |
| 81 | |
| 82 | def __init__(self, config: VTPConfig): |
| 83 | super().__init__(config) |
| 84 | self.config = config |
| 85 | |
| 86 | self._init_vision_components() |
| 87 | if config.train_clip: |
| 88 | self._init_text_components() |
| 89 | |
| 90 | self.post_init() |
| 91 | |
| 92 | def _init_vision_components(self): |
| 93 | """Initialize vision encoder and related components.""" |
| 94 | config = self.config |
| 95 | |
| 96 | # Vision encoder |
| 97 | self.trunk = DinoVisionTransformerWithBottleneck( |
| 98 | img_size=config.image_size, |
| 99 | patch_size=config.vision_patch_size, |
| 100 | embed_dim=config.vision_embed_dim, |
| 101 | depth=config.vision_depth, |
| 102 | num_heads=config.vision_num_heads, |
| 103 | ffn_ratio=config.vision_mlp_ratio, |
| 104 | ffn_layer=config.vision_ffn_layer, |
| 105 | norm_layer=config.vision_norm_layer, |
| 106 | layerscale_init=config.vision_init_values, |
| 107 | use_qk_norm=config.vision_use_qk_norm, |
| 108 | vit_feature_bottleneck=config.vision_feature_bottleneck, |
nothing calls this directly
no outgoing calls
no test coverage detected