(I'm assuming infinite data, finite time for this comparison)
I was wondering why it is exactly that online learning algorithms usually perform more poorly than their offline counter-parts.. Does anyone have a good intuitions and/or theoretical reason for this?
Why does being able to look at the same sample multiple times allow for a better learning algorithm?
For many general algorithms it's very obvious why offline algorithms are better. Simply because the more data you have, the more informed of a decision you can make.
But for machine learning we are trying to learn a distribution where the more data points you have the better you will pick the distribution.
But we may pick poor decisions in the beginning that will have long-lasting consequences on our optimization.. Maybe this is the trade-off?