Why semi-gradient is used instead of the true gradient in Q-learning? I am asking a duplicated question that nobody has answered yet. In Q-learning with function approximation, the objective is to minimize MSE between the target $r + \gamma \max_{a'} Q(s',a',w)$ and the q-value function $Q(s,a,w)$, as shown below.

The update of the parameter $w$ depends on "semi-gradient", i.e., we derive the gradient w.r.t to $w$ in $l$ while treating the target as a real value.
My question is why we calculate "semi-gradient' rather than the true gradient. Is it for easiness of gradient calculation, since the target contains a $max$ operator? Is it a kind of simplification to prove some theories?
 A: 
Is it for easiness of gradient calculation, since the target contains a max operator?

That is not a problem. CNN pooling layers for example, work just fine calculating gradients through a max operation. There are some discontinuities (when there is a tie for max, the gradient is not well defined), but they are not a major practical concern (just pick which gradient to use in case of a tie).

Is it a kind of simplification to prove some theories?

It is a simplification which works well with other practical adjustments.
In optimal control problems, what you would be calculating, if you used the full gradient, is only (a sample of) the gradient of the error in the action value estimation for the current policy. It is not a direct gradient for lowest error of the optimal $\hat{q}^*$ function. Given this constraint, it is not clear whether better estimates of the gradient gain very much in a practical sense.
Even with the full gradient, Q-learning is still a self-referential bootstrapping approach, and that suffers from some weaknesses:


*

*For off-policy TD learning combined with neural networks, there is a significant issue with bootstrap bias and the positive feedback loop that can cause. Q-learning estimates can diverge because of this. Fixes for this include experience replay and using a frozen copy of the $\hat{q}$ network to calculate the TD target.

*For Q learning, maximisation bias is a problem, whereby the action chosen is more likely to have an over-estimate of its true value. This can be fixed by double Q-learning.
In both these cases, the fix typically used is to maintain separate estimates for bootstrapping value and action choice. That can mean a fully-featured DQN solution may have 2 or 4 neural networks, with a complex relationship between their parameters. Calculating the gradient wrt $\mathbf{w}$ is not so easy with those relationships, and as any $\mathbf{w'}$ (for the networks being used to calculate TD target) is not being updated directly, then the semi-gradient is actually a more accurate representation for the update when these fixes are applied.
