Foundation Models

Inference Latency

Inference latency is the time between a model receiving an observation and producing its output, which for robot policies bounds the achievable control rate and reaction time. Billion-parameter vision-language-action models can take tens to hundreds of milliseconds per forward pass, so systems mitigate latency with action chunking, asynchronous execution, quantization, distillation, and hierarchical designs that reserve large models for slow deliberation.

Why it matters for physical AI

A policy that reasons brilliantly but reacts slowly fails at dynamic tasks, so the latency-capability trade-off dictates model size, hardware selection, and architecture for every deployed learned controller.