Foundation Models

Transformer Architecture

The transformer architecture is a neural network design built on self-attention, introduced by Vaswani et al. in 2017, in which every token in a sequence attends to every other, enabling parallel training and scalable modeling of long-range dependencies. Transformers displaced recurrent networks across language and vision and now dominate robotics: policies, world models, and vision-language-action systems tokenize observations and actions into sequences the same attention machinery processes.

Why it matters for physical AI

The transformer's uniform treatment of heterogeneous tokens is what lets a single model ingest images, language, proprioception, and actions together, making it the architectural substrate of essentially all current robot foundation models.