Robot Learning

Offline-to-Online RL

Offline-to-online RL is a training regime that first pretrains a policy and value function on a static dataset with offline reinforcement learning, then continues improving them with limited online interaction. The offline phase provides a competent starting point, while the online phase corrects value errors and refines behavior; methods such as AWAC and Cal-QL address the instability that can occur at the handoff.

Why it matters for physical AI

Warm-starting from logged data slashes the real-world interaction needed to reach reliable performance, a practical recipe for improving deployed robot skills with minutes rather than days of trials.