Foundation Models

BERT

BERT (Bidirectional Encoder Representations from Transformers) is a language model from Google, introduced by Devlin et al. in 2018, that pretrains a transformer encoder with masked-token and next-sentence objectives, then fine-tunes it for downstream tasks. It established the pretrain-then-fine-tune paradigm at scale, and its bidirectional masked modeling inspired representation learners across domains, including masked autoencoders in vision and BERT-style objectives over robot trajectory tokens.

Why it matters for physical AI

The transfer learning recipe BERT proved out, generic pretraining followed by cheap task adaptation, is precisely the template robot foundation models now apply to sensorimotor data.