4
$\begingroup$

In my day to day work, I train models on data using R packages that have no extension for Bayesian priors. I will generally have a large dataset to start off with, and add new data as needed.

Any time I want to update the model, I have to train the entire thing from scratch.

Are there ways of mitigating the considerable and slowly-increasing time cost of re-training everything from scratch, when I am unable to use Bayesian priors in my model?

A couple of approaches have occurred to me. Model training generally allows for initial weights/parameters to be specified. Setting the initial weights to the weights of the previous model may be a start, but presumably you need to include the previous data, or else the model will move from the old weights to capture only the new data.

Does training old + new data using initial weights trained from old data decrease the training time appreciably? Are there any other practical considerations for dealing with this type of situation?

To narrow it down, the particular models I am looking at tend to be complicated ones for predictive (not explanatory) purposes. Neural networks, word-vectors etc. I am not using any form of regularisation.

$\endgroup$
  • $\begingroup$ This is dependant to the type of models we are using. For example in the context of regularised regression we might use ridge regression and treat the new data as just another batch to train upon (as if using a standard Stochastic Gradient Descent). In addition, maybe the addition of new data might impose size constraints. The general idea you described is quite reasonable, but for "practical considerations" a few more clarifications about the actual application will be helpful. (+1 nice question) $\endgroup$ – usεr11852 says Reinstate Monic Mar 11 at 2:10
  • $\begingroup$ Updated the question. I can see how it might be dependent on the type of model/task. With word2vec models in particular, there can be a large number of individual words that will be present in the initial weights (from a large training run performed earlier) that will not be present in the training data. $\endgroup$ – Ingolifs Mar 12 at 1:07
0
$\begingroup$

The concept I feel you are talking about is Online learning or Incremental Learning(you can find quite a lot of literature with these terms). This is done by Vowpal Wabbit library where you can train a Regression/Classification where both are probabilistic models. This way you can actually train the model first with large data and keep on updating the model as you obtain new data. If something sticks out with new data you can reset the learning rate to a higher amount and begin training thereby to capture the new distribution which was not present with the old training dataset. Otherwise, keep the learning rate as it is to not induce variance in the model. For more practical experiences with Neural Networks, you can refer to concepts related to transfer learning.

Training old+new data using initial weights should technically be faster if both datasets are quite same since you will converge way faster, thinking mathematically ( since it has been seen that all local minima's neural network are equally good). But trying to generalize this point -

  • It depends on the size of the new data
  • It also depends on how different these 2 datasets are in terms of information
  • It also depends on your model complexity.
$\endgroup$
  • 1
    $\begingroup$ Hey! I would really like to know the reason of downvotes so I could improve upon my answer. $\endgroup$ – Axelius Mar 18 at 7:27
-1
$\begingroup$

Just an idea:

depending on your loss function, maybe it is possible to include an additional loss term for moving away from your trained model, depending on the number of new samples you have.

something like:

$\mathcal{L}(\phi) = loss + \frac{1}{N_{new}} D_{KL}(p_\phi^i ||p^{i+1}_\phi) $

But of course this is also a kind of regularization, which you want to avoid, if I understood correctly?

sorry, this is more a comment than an answer, but I don't have the reputation to comment...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.