Temporal difference definition (Reinforcement Learning) Reading for instance Szepesvari or this : i struggle to understand the rationale behind the temporal-difference definition 
$\delta_{t}=R_{t}+\gamma V_{k}(x_{t+1})-V_{k}(x_{t})$
with the notation from the second reference page 33
or
$\delta_{t}=r(x_{t},a_{t})+V(x_{t+1})-V(x_{t})$
with the notation from the second reference page 12.
Why are we adding the reward to a difference between value function evaluation at two time steps ? I would expect this new reward to be included in $V(x_{t+1})$.
 A: By definition, the true value function, $V(x_t)$ is defined to be $V(x_t) = \max_{a_t} \mathbb E[R_t + \gamma V(x_{t+1})]$ where the expectation is over the uncertainty in the reward process and the transition function. Then assuming an optimal policy, a single realization of $R_t + \gamma V(x_{t+1})$ is an unbiased estimator if $V(x_t)$. The temporal difference is therefore how much the current estimate of the value function deviates from the newest sample, and can be thought of as a noisy signal for which direction to update beliefs in in light of as the latest data.
Of course, during learning, the policy adopted is not optimal, so for small $t$, $R_t + \gamma V(x_{t+1})$ will not be a good estimate of $V(x_t)$, but the above gives some intuition for why this difference might be an object of interest (it will get better at unbiasedly estimating the value function as our learner begins acting more and more optimally). The demonstration of this requires some more high powered math (in particular, the Banach contraction mapping principle).
