2021年3月29日 · We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks.

看下这个ViViT的实现： 1) 先是通过：temporal_encode，将视频编码好， 2) 通过Encoder： 加位置编码和多少重复的layer

先看一下ViT (Image)的公式定义： 输入为 x_ {i}\in \mathbb {R}^ {h\times w}, 经过线性映射后，变换为一维tokens， z_ {i} \in \mathbb {R^ {d}}。 输入的编码表达如下: {z} = \left [z_ {cls}, {E}x_ {1}, {E}x_ {2},..., {E} {x_ {N}} \right] + {p} \\ 这里的 {E} 是2d卷积。如下图最左边结构所示，添加一个可学习的token, z_ {cls},用于表达最后分类之前的特征。 这里， {p} \in \mathbb {R}^ {N\times d} 表示为位置编码，加上输入token，用来保留位置信息。

2021年11月30日 · 视觉Transformer (ViT)采用了Transformer架构，以最小的变化处理二维图像。

The Vivit model was proposed in ViViT: A Video Vision Transformer by Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid. The paper proposes one of the first successful pure-transformer based set of models for video understanding.

An unofficial implementation of ViViT. We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatiotemporal tokens from the input video, which are then encoded by a series of transformer layers.

2024年1月1日 · The ViViT model's architecture seeks to extend the success of transformer-based architecture for image classification, the Vision Transformer (ViT), into the video space. The ViViT model treats inputted videos as spatio-temporally encoded tokens, representing chunks of the video at different times and areas of the frame.

2025年1月7日 · 视频视觉Transformer（ViViT）是一种新型的视频分类模型，它使用Transformer架构来处理视频数据。与传统的卷积神经网络不同，ViViT使用自注意力机制来捕捉视频中的时空关系。这种方法可以更好地处理视频中的长期依赖关系，并且可以在不使用卷积的情况 …

Inspired by ViT, and the fact that attention-based ar-chitectures are an intuitive choice for modelling long-range contextual relationships in video, we develop sev-eral transformer-based models for video classification.

We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers.