21ICCV # Emerging Properties in Self-Supervised Vision Transformers (DINO)

[XFeiF]

Paper
Code

Authors:
Mathilde Caron, Hugo Touvron, etc.
FBAI.

Highlights:

• A new proposed self-supervised learning method with KD: a form of knowledge distillation with no labels. Especially, it uses a different way to avoid the collapse solution, that is use the momentum teacher encoder.
• It encouraging "local-to-global" correspondences by feeding different sizes of views to student and teacher encoders.
• SSL ViT features explicitly contain the scene layout and, in particular, object boundaries, as shown in the next figure.

[XFeiF]

They look at the self-attention of the [CLS] token on the heads of the last layer. This token is not attached to any label nor supervision. These maps show that the model automatically learns class-specific features leading to unsupervised object segmentations.

SSL segmentation magic -> SSL + Transformer's [CLS] token.