I have been going through "Sutton & Barto Book: Reinforcement Learning: An Introduction", and in "Chapter 11: Off-policy Methods with Approximation", Example 11.1 briefly discusses counter example given by Tsitsiklis and Van Roy in their paper "Feature-based methods for large scale dynamic programming" (https://link.springer.com/article/10.1007/BF00114724)
What I don't understand is the difference between the usage of $ w$ and $w_{k}$ in the equations. Why are we applying $argmin$ with respect to $w$ rather than $w_k$? Are $w$ and $w_k$ not the same thing, namely the parameters of the linear function approximator at time $t$? Since we have current approximate state values $w$ and $2w$, as given by the figure, our parameter $w_k$ should be equal to $w$. But somehow authors decide to use $w$ and $w_k$ and minimize with respect to $w$, not $w_k$. Is this because of semi-gradient operation with an aim to differentiate only with respect to $w$ and treating expectation $\mathbb{E}_\pi[\cdots]$, a function of $w_k$, as a constant?
How do we reach to $w_{k+1} = [(6 - 4 \epsilon) / 5] \lambda w_{k}$ analytically? Can someone explain me step by step minimization procedure? As both $(w - \lambda 2 w_k)^2$ and $(2w - (1 - \epsilon) \lambda 2 w_k)^2$ are squared expressions, both can not be equal to less than zero, so they should both be zero to achieve the minimum value for the sum of squares, namely zero. If so it should be that $(w - \lambda 2 w_k)^2 = 0$ and $(2w - (1 - \epsilon) \lambda 2 w_k)^2 = 0$. So, $w = \lambda 2 w_k$, then how did we achieve $w_{k+1} = w =[(6 - 4 \epsilon) / 5] \lambda w_{k}$ ?
