Decodes a sequence of ImageBind spatial tokens from a model dictionary into an PCA representation. Args: mod_dict (dict): Model output dictionary. key (str): Key of the tokenized CLIP modality to decode. tokenizers (dict): Dictionary of tokenizers. image_siz
(mod_dict, tokenizers, key='tok_imagebind', image_size=224, patch_size=14)
| 306 | return pca_viz |
| 307 | |
| 308 | def decode_tok_imagebind(mod_dict, tokenizers, key='tok_imagebind', image_size=224, patch_size=14): |
| 309 | """ |
| 310 | Decodes a sequence of ImageBind spatial tokens from a model dictionary into an PCA representation. |
| 311 | |
| 312 | Args: |
| 313 | mod_dict (dict): Model output dictionary. |
| 314 | key (str): Key of the tokenized CLIP modality to decode. |
| 315 | tokenizers (dict): Dictionary of tokenizers. |
| 316 | image_size (int): Size of the image. |
| 317 | patch_size (int): Size of the patches. |
| 318 | """ |
| 319 | patch_size = 14 |
| 320 | n_patches = image_size // patch_size |
| 321 | img_tok = rearrange(mod_dict[key]['tensor'], "b (nh nw) -> b nh nw", nh=n_patches, nw=n_patches) |
| 322 | rec = tokenizers[get_transform_key(key)].decode_tokens(img_tok) |
| 323 | pca_viz = [pca_visualize(feat) for feat in rec] |
| 324 | pca_viz = np_squeeze(np.stack(pca_viz), axis=0) |
| 325 | return pca_viz |
| 326 | |
| 327 | def decode_tok_dinov2_global(mod_dict, tokenizers, key='tok_dinov2_global'): |
| 328 | """ |
no test coverage detected