I came across this equation in the proof of the compatible function approximation theorem:


My question is related to the first 2 equations:

concerned equation

To my understanding, $Q(s,a;w)$ is a bias estimate of $Q^{\pi}(s,a)$, and the initial value of $w$ will always have influence on $Q(s,a;w)$ (as suggested in this answer: Reinforcement Learning - Why are actor critic methods biased?).

So this means that $Q(s,a;w)$ will never be able to reach $Q^{\pi}(s,a)$. However in the equation above, it say that $\epsilon$ has to be minimized. But I don't know if the proof means $\epsilon$ has to be minimized to exactly 0 or just close to:

if $\epsilon$ minimized to exactly 0:

then there is no bias anymore, $Q(s,a;w)$ = $Q^{\pi}(s,a)$, which contradict with the fact that $Q(s,a;w)$ is a bias estimate.

if $\epsilon$ minimized to close to 0:

then there will be an error term in rest of the equations in the proof, this contradicts with the original objective, which is to remove bias.

Both case doesn't make any sense to me, I am sure that I got something wrong, but I don't know where.

The whole proof: https://www.slideshare.net/cover_drive/policy-gradient-theorem


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.