Robot Learning

Bellman Equation

The Bellman equation is the recursive relationship stating that the value of a state equals the expected immediate reward plus the discounted value of the successor state, formulated by Richard Bellman in the 1950s as the foundation of dynamic programming. Its fixed point defines the optimal value function, and temporal-difference methods including Q-learning and the critics of actor-critic algorithms are constructed as stochastic approximations of Bellman backups.

Why it matters for physical AI

Every value-based component in modern robot reinforcement learning, from locomotion critics to offline Q-functions, is trained against Bellman targets, whose approximation errors drive the stability tricks these systems require.