Robot Learning

Value Iteration

Value iteration is a dynamic programming algorithm that computes the optimal value function of a Markov decision process by repeatedly applying the Bellman optimality backup to every state until the values converge, after which the optimal policy is extracted greedily. It requires a known transition model and an enumerable state space, so in robotics it is applied to discretized problems such as grid-based navigation and serves as the conceptual foundation for approximate and learned variants.

Why it matters for physical AI

Grid-world value iteration still powers practical costmap-based planners, and its fixed-point structure underlies deep RL methods, making it essential background for understanding how learned critics and planners behave.