4
$\begingroup$

I was reading those papers by Facebook and Google on Click-Through Rate predictions and something kept bothering me.

I understand that to fully take advantage of data freshness or simply because we work at massive scale it is interesting to learn/update our model in an online/streaming fashion. I also understand that using a decreasing learning rate might yield better models.

But I don't understand intuitively how these two can mesh well without human interaction: in this setting we're talking about billions of training examples per day and with a decreasing learning rate at iteration $t$ of the form $\alpha/\sqrt{t}$, that learning rate would be so small at the end of day that it would kill any chance of learning anything forever.

So what happens next in practice? Do we have to restart the learning rate at $\alpha/\sqrt{1}$ with the last set of coefficients (having a logistic regression in mind)? Is my understand of "iteration" wrong? Is my intuition simply wrong?

$\endgroup$
2
$\begingroup$

in this setting we're talking about billions of training examples per day and with a decreasing learning rate at iteration $t$ of the form $\alpha/\sqrt{t}$, that learning rate would be so small at the end of day that it would kill any chance of learning anything forever.

From the first paper you cite {1}:

We lower bound the learning rates by 0.00001 for continuous learning


References:

$\endgroup$
  • $\begingroup$ Completely forgot about that... Thank you very much! $\endgroup$ – Florent Moiny Jan 1 '17 at 21:20
  • $\begingroup$ Wait; this doesn't seem to answer the deeper question here - why can we just start using a constant learning rate a couple days in? That breaks all the proofs for online learning convergence! $\endgroup$ – VF1 Jan 1 '17 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.