4
$\begingroup$

Optimization algorithms like Adagrad and ADAM decay your learning rate over time. To me this sounds like a bad idea for online training since you're always getting new data as opposed to retraining on the same data for multiple epochs in offline.

Suppose I could use Adagrad or ADAM for online training, would the learning rate I find using grid search for offline training be suitable for online training? I'd imagine not.

$\endgroup$
1
$\begingroup$

If I understood correctly Adagrad will decay the learning rate as there's a matrix $G_t=\sum g_\tau g_\tau^T$ whose value is always increasing, while in Adam a similar matrix is estimated by moving average to avoid such decay, and it seems the idea of moving average fits well the context of online learning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.