1
$\begingroup$

For one step Temporal Difference Learning does the learning rate parameter $\alpha$ require the Robbins-Monro condition?

$$ \sum \alpha_t =\infty \quad \text{and}\quad \sum \alpha^{2}_t <\infty$$

$\endgroup$
1
$\begingroup$

The original proof for TD(0) given in [1] is a proof for convergence in the mean. Since this is not convergence in probability, there is no need for the Robbins-Monro conditions. However, there is a proof for convergence in probability given in [2] that does require the Robbins-Monro conditions to hold.

[1] Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine learning, 3(1), 9–44.

[2] Dayan, P. (1992). The convergence of td ($\lambda$) for general $\lambda$. Machine learning, 8(3-4), 341–362.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.