Decreasing learning rate with online/streaming learning

Question

I was reading those papers by Facebook and Google on Click-Through Rate predictions and something kept bothering me.

I understand that to fully take advantage of data freshness or simply because we work at massive scale it is interesting to learn/update our model in an online/streaming fashion. I also understand that using a decreasing learning rate might yield better models.

But I don't understand intuitively how these two can mesh well without human interaction: in this setting we're talking about billions of training examples per day and with a decreasing learning rate at iteration $t$ of the form $\alpha/\sqrt{t}$, that learning rate would be so small at the end of day that it would kill any chance of learning anything forever.

So what happens next in practice? Do we have to restart the learning rate at $\alpha/\sqrt{1}$ with the last set of coefficients (having a logistic regression in mind)? Is my understand of "iteration" wrong? Is my intuition simply wrong?

Answer 1

in this setting we're talking about billions of training examples per day and with a decreasing learning rate at iteration $t$ of the form $\alpha/\sqrt{t}$, that learning rate would be so small at the end of day that it would kill any chance of learning anything forever.

From the first paper you cite {1}:

We lower bound the learning rates by 0.00001 for continuous learning

---

References:

• {1} He, Xinran, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi et al. "Practical lessons from predicting clicks on ads at facebook." In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising, pp. 1-9. ACM, 2014. https://pdfs.semanticscholar.org/daf9/ed5dc6c6bad5367d7fd8561527da30e9b8dd.pdf