I'm having difficulty finding any explanation as to why standard Q-learning tends to overestimate q-values (which is addressed by using double Q-learning). The only sources I have found don't really explain exactly why this overestimation occurs.
For example, the Wikipedia article on Q-learning says:
Because the maximum approximated action value is used in the Q-learning update, in noisy environments Q-learning can sometimes overestimate the actions values, slowing the learning.
What does this mean? I understand Q-learning, but not the above. Why does the use of the maximum q-value cause overestimation?