Foundation Models

Knowledge Distillation

Knowledge distillation is the training of a smaller student model to reproduce the outputs or intermediate representations of a larger teacher, introduced by Hinton, Vinyals, and Dean in 2015 using softened output distributions. In robotics it compresses large vision-language-action models for edge deployment and appears as teacher-student policy distillation, where a privileged-observation teacher trained in simulation supervises a vision-based student.

Why it matters for physical AI

Distillation reconciles the scale that makes foundation policies capable with the latency and power budgets of onboard compute, and privileged-teacher schemes are a standard route to deployable sim-trained controllers.