Model selection in offline vs. online learning

Question

I've been trying to learn more about online learning lately (it's absolutely fascinating!), and one theme that I haven't been able to get a good grasp on is how to think about model selection in offline versus online contexts. Specifically, suppose we train a classifier $S$ offline, based on some fixed data set $D$. We estimate its performance characteristics via cross-validation, say, and we select the best classifier this way.

This is what I've been thinking about: how, then, do we go about about applying $S$ to an online setting? Can we assume that the best $S$ found offline will also perform well as an online classifier? Does it make sense to gather some data to train $S$, then take that same classifier $S$ and "operationalize" it in an online setting with he same parameters found on $D$, or might another approach be better? What are the caveats in these cases? What are the key results here? And so forth.

Anyway, now that's out there, I guess what I'm looking for is some references or resources that will help me (and hopefully others, who have been thinking about this kinda thing!) make the transition from thinking solely in offline terms, and develop the mental framework to think about the issue of model selection and these questions in a more coherent way as my reading progresses.

Answer 1

Obviously, in a streaming context you cannot split your data into train and test sets to perform cross-validation. Using only the metrics calculated on the initial train set sounds even worse, as you assume that your data changes and your model will adapt to the changes--that is why you are using the online learning mode in the first place.

What you could do is to use the kind of cross-validation that is used in time-series (see Hyndman and Athanasopoulos, 2018). To assess accuracy of time-series models, you could use a sequential method, where the model is trained on $k$ observations to predict on the $k+1$ "future" timepoint. This could be applied one point at a time, or in batches, and the procedure is repeated until you have traversed all your data (see the figure below, taken from Hyndman and Athanasopoulos, 2018).

At the end, you somehow average (usually arithmetic mean, but you could use something like exponential smoothing as well) the error metrics to obtain the overall accuracy estimate.

In an online scenario this would mean that you start at timepoint 1 and test on timepoint 2, next re-train on timepoint 2, to test on timepoint 3 etc.

Notice that such cross-validation methodology lets you account for the changing nature of your models performance. Obviously, as your model adapts to the data and the data may change, you would need to monitor the error metrics regularly: otherwise it wouldn't differ much from using fixed-size train and test sets.