# When implementing TD(λ) with eligibility traces, why do we update towards the same 𝛿?

When using $TD(\lambda)$ in a finite state-action space, the update looks like

$$\forall s,a \in S, A(Q(s, a) = Q(s, a) + \alpha \delta e_t(s, a))$$

where

$$\delta = R + \gamma Q(s_{t + 1}, a_{t + 1}) - Q(s_t, a_t)$$

This process updates every state-action towards the delta between the observed TD(0) return and the current state.

But looking at $TD(\lambda)$ from the forward view,

$$\delta = R + (1 - \lambda)\sum_{n = 1}^{\infty} \lambda^{n - 1} \gamma^n Q(s_{t + n}, a_{t + n}) - Q(s_t, a_t)$$

where the delta is between the $\lambda$-return and the current state. In the backwards view it seems like this "current" state is equivalent to the state-action being updated, not the current state of the agent.

Shouldn't the update be

$$\forall s,a \in S, A(Q(s, a) = Q(s, a) + \alpha \delta(s, a) e_t(s, a))$$

where

$$\delta(s, a) = R + \gamma Q(s_{t + 1}, a_{t + 1}) - Q(s, a)$$

This would mean computing $\delta(s, a)$ for every state-action along with the eligibility trace.

## 1 Answer

One update step of backward-TD is not equivalent with one update of forward. Rather, the sum of them are. If you keep going, the value at a particular state will be updated with the weighted deltas of all the future states. As a particularly illustrative case, if you set lambda=1, the V(s_t+1) of one delta cancels with the V(s_t) at the next time step and you end up just adding the rewards, yielding the monte carlo update