2022年11月17日 · To investigate strategies for refining the CLIP-targeted MIM, we study two critical elements in MIM, i.e., the supervision position and the mask ratio, and reveal two interesting perspectives, relying on our developed simple pipeline, context autodecoder with CLIP target (CAE v2).

我们最近的工作 “ Context Autoencoder for Self-Supervised Representation Learning ”，提出一种新的 MIM 方法 CAE，通过对 “表征学习” 和 “解决 pretext task” 这两个功能做尽可能的分离，使得 encoder 学习到更好的表征，从而在下游任务实现了更好的泛化性能。 我们尝试回答如下几个问题： MIM 方法中，网络结构的哪个部分是学习表征的，哪个部分是解决 pretext task？ 为什么之前典型的 contrastive learning 方法，在下游任务 (例如检测、分割) 上只能取得跟 supervised …

2023年10月5日 · CAE v2 is an improved variant of CAE (Chen et al., 2023), applying the CLIP latent on two pretraining tasks, i.e., visible latent alignment and masked latent alignment. Visible latent alignment directly mimics the visible latent representations from the encoder to the corresponding CLIP latent, which is beneficial for facilitating model ...

2023年10月11日 · This is a PyTorch implementation of CAE: Context AutoEncoder for Self-Supervised Representation Learning. State-of-the-art MIM performance. Results in the paper are successfully reproduced. 2023/11/28: We release the training code of CAE v2. Please refer to project/CAEv2/README.md.

2022年2月7日 · We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded...

To conduct the study, we develop a simple MIM pipeline, i.e., context autoencoder with CLIP target (CAE v2). We will first analyze how the supervision position and the mask ratio influence the perfor-mance of CAE v2.

With our simple pipeline CAE v2, we reveal two new insights: i) the feature distillation supervision on visible patches can achieve remarkable performance; ii) the optimal mask ratio is positively correlated to the model size.

This is the official repository with PyTorch implementation of CAE v2: Context Autoencoder with CLIP Latent Alignment. Release a serious of pre-trained models in CAE v2, including CAEv2-Tiny, CAEv2-Small, CAEv2-Base and CAEv2-Large. Please refer to Google Drive to download. Download the pretrained clip model.

2022年11月17日 · To investigate strategies for refining the CLIP-targeted MIM, we study two critical elements in MIM, i.e., the supervision position and the mask ratio, and reveal two interesting perspectives,...

本文提出了一个MIM方法，context autoencoder（CAE），它首先将图像分为可见块和掩码块，利用可见块来预测掩码块的表征，并通过一致性约束将预测的表征限制在编码器的表征空间中。 最后，通过解码器将预测的掩码块表征映射到前置任务的目标上。 与先前的方法比，本文的方法将表征学习（encoding）和前置任务拆分，增强了模型表征能力的同时，也提升了在下游任务的性能。 文章还解释了为什么对比预训练和监督预训练表现相似，以及为什么MIM可以表现得更好。 …