2
$\begingroup$

In Richard S. Sutton and Andrew G. Barto's book on reinforcement learning on page 156 it says:

Maximization bias occurs when estimate the value function while taking max on it (that is what Q learning do), and maximization may not take on the true value which may introduce bias.

  1. Why can double Q learning solve this problem, and what is the proof for that?
  2. Does maximization bias always underestimate or always overestimate the true value? Why?
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ always overestimates, because of the max() operation $\endgroup$
    – Gabizon
    Commented Aug 20, 2019 at 20:31
  • $\begingroup$ Maybe you refer to Jensen inequality, but why it's convex and convex respect to which variable? $\endgroup$
    – yi li
    Commented Aug 21, 2019 at 3:42

1 Answer 1

Reset to default
2
$\begingroup$

One way to view the problem of maximizing bias is that it is due to using the same samples both to determine the maximizing action and to estimate its value. So, we decouple in two policies Q1(a) and Q2(a) and choose, say Q1(a) to choose maximizing action and Q2(a) to estimate its value.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.