Robot Learning

Conservative Q-Learning (CQL)

Conservative Q-Learning is an offline reinforcement learning algorithm from Kumar et al. (2020) that penalizes Q-values of actions outside the dataset distribution, learning a lower bound on the true value function to prevent the overestimation that plagues offline RL. The added regularizer pushes down Q-values for out-of-distribution actions while pushing up values for dataset actions, yielding policies that improve on the data without exploiting extrapolation errors.

Why it matters for physical AI

Robots accumulate large logs of prior experience that cannot be safely extended by online trial and error; conservative offline RL methods extract improved policies from such fixed datasets without dangerous exploration.