← back to publications
GViT: Representing Images as Gaussians for Visual Recognition
preprint

GViT: Representing Images as Gaussians for Visual Recognition

Jefferson Hernandez, Ruozhen He, Guha Balakrishnan, Alexander C. Berg, Vicente Ordonez.
arXiv:2506.23532
paper arxiv pdf raw bibtex

abstract

We introduce GVIT, a classification framework that abandons conventional pixel or patch grid input representations in favor of a compact set of learnable 2D Gaussians. Each image is encoded as a few hundred Gaussians whose positions, scales, orientations, colors, and opacities are optimized jointly with a ViT classifier trained on top of these representations. We reuse the classifier gradients as constructive guidance, steering the Gaussians toward class-salient regions while a differentiable renderer optimizes an image reconstruction loss. We demonstrate that by 2D Gaussian input representations coupled with our GVIT guidance, using a relatively standard ViT architecture, closely matches the performance of a traditional patch-based ViT, reaching a 76.9% top-1 accuracy on Imagenet-1k using a ViT-B architecture.

citation

@article{hernandezgvit,
  title = {GViT: Representing Images as Gaussians for Visual Recognition},
  author = {Hernandez, Jefferson and He, Ruozhen and Balakrishnan, Guha and Berg, Alexander C. and Ordonez, Vicente},
  journal = {arXiv preprint arXiv:2506.23532},
  url = {https://arxiv.org/abs/2506.23532},
}