This repository contains the code and datasets for our ICCV-W paper 'Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts' Resources

2024年6月17日 · 我们的 少样本适配器CLIP-A-self学会了从GPT生成的集合中选择最佳的VDT信息，并在Base-to-New设置中提高了少样本域迁移性能，即使生成的文本质量下降也是如此。

2024年2月28日 · 提出统一的时空掩码建模机制，使 VDT 能够处理多种视频生成任务，实现了技术的广泛应用。VDT 灵活的条件信息处理方式，如简单的 token 空间拼接，有效地统一了不同长度和模态的信息。

VDT通过在 token 级别拼接条件帧(潜在特征)和噪声帧来实现这一点，然后将其输入到 VDT 中 接下来，他们将 VDT 的输出帧序列分割，并使用预测的帧进行扩散过程，如上图(b)所示

2023年7月21日 · In this work, we show that GPT-4 can be used to generate text that is visually descriptive and how this can be used to adapt CLIP to downstream tasks. We show considerable improvements in 0-shot transfer accuracy on specialized fine-grained datasets like EuroSAT (~7%), DTD (~7%), SUN397 (~4.6%), and CUB (~3.3%) when compared to CLIP's default ...

ensembling the VDT sentences reduce CLIP’s performance sensitivity to small changes in the prompt. We show per-formance improvements over vanilla CLIP with the default prompt on 12 datasets with an average improvement of 2% and even better improvements in fine-grained datasets like EuroSAT (∼7%), DTD (∼7%), SUN397 (∼4.6%), and CUB (∼3 ...

CLIP是一个由图像编码器和文本编码器的双流网络。如果编码器是ViT(图像)或者BERT(文本), 那么<cls>位置上的嵌入向量被用作表示整个图像或文本的特征向量。

main.sh is the script for running default clip adapter. Please refer to b2n_adapters.sh for the scripts for all shots and all datasets (with tuned residual ratio) for CLIP-A-self in the base 2 new setting.

Official implementation of 'CLIP-Adapter: Better Vision-Language Models with Feature Adapters'. CLIP-Adapter is a drop-in module designed for CLIP on few-shot classfication tasks. CLIP-Adapter can improve the few-shot classfication of CLIP with very simple design. We utilize the code base of CoOp.

2023年7月21日 · In this work, we show that GPT-4 can be used to generate text that is visually descriptive and how this can be used to adapt CLIP to downstream tasks. We show considerable improvements in 0-shot transfer accuracy on specialized fine-grained datasets like EuroSAT (~7%), DTD (~7%), SUN397 (~4.6%), and CUB (~3.3%) when compared to CLIP's default ...