Practical realities of updating a trained model with new data

Question

In my day to day work, I train models on data using R packages that have no extension for Bayesian priors. I will generally have a large dataset to start off with, and add new data as needed.

Any time I want to update the model, I have to train the entire thing from scratch.

Are there ways of mitigating the considerable and slowly-increasing time cost of re-training everything from scratch, when I am unable to use Bayesian priors in my model?

A couple of approaches have occurred to me. Model training generally allows for initial weights/parameters to be specified. Setting the initial weights to the weights of the previous model may be a start, but presumably you need to include the previous data, or else the model will move from the old weights to capture only the new data.

Does training old + new data using initial weights trained from old data decrease the training time appreciably? Are there any other practical considerations for dealing with this type of situation?

To narrow it down, the particular models I am looking at tend to be complicated ones for predictive (not explanatory) purposes. Neural networks, word-vectors etc. I am not using any form of regularisation.

Answer 1

The concept I feel you are talking about is Online learning or Incremental Learning(you can find quite a lot of literature with these terms). This is done by Vowpal Wabbit library where you can train a Regression/Classification where both are probabilistic models. This way you can actually train the model first with large data and keep on updating the model as you obtain new data. If something sticks out with new data you can reset the learning rate to a higher amount and begin training thereby to capture the new distribution which was not present with the old training dataset. Otherwise, keep the learning rate as it is to not induce variance in the model. For more practical experiences with Neural Networks, you can refer to concepts related to transfer learning.

Training old+new data using initial weights should technically be faster if both datasets are quite same since you will converge way faster, thinking mathematically ( since it has been seen that all local minima's neural network are equally good). But trying to generalize this point -

• It depends on the size of the new data
• It also depends on how different these 2 datasets are in terms of information
• It also depends on your model complexity.

Answer 2

Just an idea:

depending on your loss function, maybe it is possible to include an additional loss term for moving away from your trained model, depending on the number of new samples you have.

something like:

$\mathcal{L}(\phi) = loss + \frac{1}{N_{new}} D_{KL}(p_\phi^i ||p^{i+1}_\phi) $

But of course this is also a kind of regularization, which you want to avoid, if I understood correctly?

sorry, this is more a comment than an answer, but I don't have the reputation to comment...