When using $TD(\lambda)$ in a finite state-action space, the update looks like

$$ \forall s,a \in S, A(Q(s, a) = Q(s, a) + \alpha \delta e_t(s, a))$$


$$\delta = R + \gamma Q(s_{t + 1}, a_{t + 1}) - Q(s_t, a_t)$$

This process updates every state-action towards the delta between the observed TD(0) return and the current state.

But looking at $TD(\lambda)$ from the forward view,

$$\delta = R + (1 - \lambda)\sum_{n = 1}^{\infty} \lambda^{n - 1} \gamma^n Q(s_{t + n}, a_{t + n}) - Q(s_t, a_t)$$

where the delta is between the $\lambda$-return and the current state. In the backwards view it seems like this "current" state is equivalent to the state-action being updated, not the current state of the agent.

Shouldn't the update be

$$ \forall s,a \in S, A(Q(s, a) = Q(s, a) + \alpha \delta(s, a) e_t(s, a))$$


$$\delta(s, a) = R + \gamma Q(s_{t + 1}, a_{t + 1}) - Q(s, a)$$

This would mean computing $\delta(s, a)$ for every state-action along with the eligibility trace.


One update step of backward-TD is not equivalent with one update of forward. Rather, the sum of them are. If you keep going, the value at a particular state will be updated with the weighted deltas of all the future states. As a particularly illustrative case, if you set lambda=1, the V(s_t+1) of one delta cancels with the V(s_t) at the next time step and you end up just adding the rewards, yielding the monte carlo update


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.