Foundation Models

Transformer Visual Backbone

A transformer visual backbone is a vision encoder based on the Vision Transformer (ViT), which splits an image into patches, embeds them as tokens, and processes them with self-attention to produce visual representations. Pretrained ViT variants, including CLIP encoders, DINOv2, and SigLIP, serve as the perception front-ends of most vision-language and vision-language-action models, with robotics-specific pretraining efforts like R3M and VC-1 targeting manipulation-relevant features.

Why it matters for physical AI

The visual backbone determines what a robot policy can perceive before any control learning begins, and reusing internet-pretrained encoders is the main mechanism by which semantic knowledge from the web reaches physical robots.