0
$\begingroup$

In TD/SARSA-lambda, eligibility traces are decayed after each step by multiplying by the discount rate and the trace-decay parameter.

eligibility trace update

I understand that:

  • The discount rate is used to reduce the value of future actions relative to a state.
  • An eligibility trace should decay over time, as it represents how "responsible" a state-action pair is for an obtained reward.

What I don't understand why the trace-decay parameter is required, given that multiplying by the discount rate after each step will cause it to decay over time anyways. What does the trace-decay parameter add to the algorithm?

$\endgroup$

1 Answer 1

2
$\begingroup$

The discount factor $\gamma$ is applied to the scalar sum of future rewards. It changes how much a single measure of future reward will contribute to an update step.

The trace decay $\lambda$ is applied to the weight update vector. It changes which weights receive an update, and in which proportion.

When updating the trace it must also take into account the discount factor, but the value function update itself uses $\gamma$ but not $\lambda$. In addition, the upadte step for the eligilibity vector is different to and separate from the value function estimate updates, so it will evolve differently as the agent progresses through the state trajectory on each time step.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.