Robot Learning

Reward Shaping

Reward shaping is the addition of auxiliary reward terms to a sparse task objective to guide exploration, such as rewarding distance reduction toward a goal or intermediate subtask completion. Ng, Harada, and Russell (1999) proved that potential-based shaping, where the bonus is a difference of a potential function over states, preserves the optimal policy, whereas arbitrary shaping can change what behavior is optimal and invite reward hacking.

Why it matters for physical AI

Shaping is often what makes sparse-reward manipulation and locomotion tasks learnable at all, but the potential-based result is a standing warning that convenient bonuses can quietly redefine the task.