Why in $Q$-Learning, policy $\pi$ is evaluated through another policy $u$? I've been watching David Silver's courses about Reinforcement Learning. According to his lectures, policy $\pi$ is evaluated by evaluating another policy $\mu$.
But I cannot understand: why is it so? I would really appreciate if somebody could explain it to me.
 A: In Monte Carlo and TD, we have to assure that all combinations of actions and states are visited an infinite number of times.
In principle, there are two ways:


*

*on-policy - you work with one policy only which implicitly combines the exploration and exploitation, e.g. a form of $\epsilon$-greedy

*off-policy - you are trying to find the best (or target) policy, but you explore your environment with another, behavior policy 


The distinguishing these two policies is a very powerful tool because of two main reasons:


*

*You can use historical data that were not optimal, generate e.g. by a human operator.

*You can stay focused on the deterministic target. If you get a randomized policy from an on-policy method, will engineers who shall integrate your controller in a broader system trust you?

A: It's hard to say exactly without any context, however, I believe, it's related to what Karel says in his answer.
$Q$-learning is off-policy since you are greedy when you compute a TD error: you take $\max_{a'}{Q(s',a')}$. 
However, when you do the rollout, you are not greedy all the time, you do exploratory actions (e.g. $\epsilon$-greedy).
