How to achieve sota performance as shown in Table 2

[MichaelYu781]

Hi,

Thanks for this impressive work

I have a question regarding the Linear Probing performance. I noticed that the results reported in Table 2 are significantly better than the curves shown in Figure 5(a).

I am aware that Table 2 utilizes a symmetric ViT (Pixel Decoder) instead of the lightweight (4-layer) decoder. My understanding is that a symmetric/heavier decoder is primarily beneficial for reconstruction quality (rFID). However, it is surprising to see it lead to such a massive improvement in Linear Probing (representation quality) as well.

Does the symmetric decoder alone account for this performance gap?

Could you kindly clarify the training setup for Table 2? Specifically, how much data and how much training data was required?

[JingfengYao]

Thank you for your question.

It doesn't. Your observation is very keen. For all linear probing experiments except those in Table 2, we utilized only the last-layer features to directly measure the semantic properties of the latents that will be used in DiT.

However, in Table 2, when comparing our model with previous representation-focused ViTs, we adopted the same DINO-style feature augmentation strategy to ensure a fair comparison. Based on our past experience, scaling up the pixel decoder (rather than the DINO or CLIP heads) yields a relatively minimal impact on the understanding performance of VTP.

You can find the implementation details in the following script.