I have been going through "Sutton & Barto Book: Reinforcement Learning: An Introduction", and in "Chapter 11: Off-policy Methods with Approximation", Example 11.1 briefly discusses counter example given by Tsitsiklis and Van Roy in their paper "Feature-based methods for large scale dynamic programming" (https://link.springer.com/article/10.1007/BF00114724)

  1. What I don't understand is the difference between the usage of $ w$ and $w_{k}$ in the equations. Why are we applying $argmin$ with respect to $w$ rather than $w_k$? Are $w$ and $w_k$ not the same thing, namely the parameters of the linear function approximator at time $t$? Since we have current approximate state values $w$ and $2w$, as given by the figure, our parameter $w_k$ should be equal to $w$. But somehow authors decide to use $w$ and $w_k$ and minimize with respect to $w$, not $w_k$. Is this because of semi-gradient operation with an aim to differentiate only with respect to $w$ and treating expectation $\mathbb{E}_\pi[\cdots]$, a function of $w_k$, as a constant?

  2. How do we reach to $w_{k+1} = [(6 - 4 \epsilon) / 5] \lambda w_{k}$ analytically? Can someone explain me step by step minimization procedure? As both $(w - \lambda 2 w_k)^2$ and $(2w - (1 - \epsilon) \lambda 2 w_k)^2$ are squared expressions, both can not be equal to less than zero, so they should both be zero to achieve the minimum value for the sum of squares, namely zero. If so it should be that $(w - \lambda 2 w_k)^2 = 0$ and $(2w - (1 - \epsilon) \lambda 2 w_k)^2 = 0$. So, $w = \lambda 2 w_k$, then how did we achieve $w_{k+1} = w =[(6 - 4 \epsilon) / 5] \lambda w_{k}$ ?

enter image description here


1 Answer 1


You are minimizing the sum of squares. This is larger than minimizing each square, which is how you got zero. Take the derivative with respect to w and set this equal to zero to get 11.10.

You then solve an equation that is linear in w:

  0 = d/dw (w+a)^2 + (2w + b)^2  = 2[ w+a + 2(2w+b) ]
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Aug 18, 2023 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.