Why is temporal difference learning biased in reinforcement learning? When I learn reinforcement learning from David Silver's online video, I saw "the objective of TD learning, $r_t + \gamma V(s_{t+1})$ is a biased target for learning value function. " I know the definition of "biased", but I'm not sure why TD learning is biased. The followings are my questions:


*

*Under the framework of functional approximation for value function, is it because now the parameters of $V(s_{t+1})$ are not correct value during training, hence the current value of $V(s_{t+1})$ could be totally meaningless which causes it to be biased?

*In that case, suppose the parameters will converge at the end of training procedure. In that case, is $r + V(s_{t+1})$ still biased or not?

*For tabular cases, is $r + V(s_{t+1})$ still biased? I think it is now an unbiased estimator of value function. Suppose the process is episodic with total number of $T$ steps. The value function at the last time step $V(s_T)$ is clearly unbiased. This causes the estimation of $V(s_{T-1})$ to be unbiased. We can repeat the process to prove for any $t$, $V(s_t)$ is unbiased. Is this correct?
 A: 1.question: You are right, that's the reason why it's biased.
2.question: Consider simple TD update rule
\begin{equation}
V(s) = V(s) + \alpha[R + \gamma V(s') - V(s)]
\end{equation}
let's assume for simplicity that $\gamma = 1$, that reward $R$ is deterministic and that you know the correct value of $V(s')$. Then recursively we have
\begin{align}
V_{n+1}(s) &= V_n(s) + \alpha[R + V(s') - V_n(s)] \\
&= (1-\alpha)V_n(s) + \alpha(R + V(s'))\\
&=(1 - \alpha)[V_{n-1}(s) + \alpha[R + V(s') - V_{n-1}(s)]] + \alpha(R + V(s'))\\
&= \ldots \\
&= (1-\alpha)^n V_1(s) + \sum_{i=1}^n \alpha(1 - \alpha)^{i-1} (R + V(s'))
\end{align}
so the starting value before any updates was random biased value $V_1(s)$ and it will keep affecting value $V_{n+1}(s)$ after $n$ updates but its influence will be $0$ as $n \rightarrow \infty$ because $(1- \alpha)^n \rightarrow 0$. So in conclusion it will still be biased but it will decrease with time, and since $V(s_{T})$  will be eventually unbiased then other states should converge to true values as well.
3.question: same analysis applies like for the above case since the starting state is also initialized to random value and $V(s_T)$ will be eventually unbiased.
