In my day to day work, I train models on data using R packages that have no extension for Bayesian priors. I will generally have a large dataset to start off with, and add new data as needed.
Any time I want to update the model, I have to train the entire thing from scratch.
Are there ways of mitigating the considerable and slowly-increasing time cost of re-training everything from scratch, when I am unable to use Bayesian priors in my model?
A couple of approaches have occurred to me. Model training generally allows for initial weights/parameters to be specified. Setting the initial weights to the weights of the previous model may be a start, but presumably you need to include the previous data, or else the model will move from the old weights to capture only the new data.
Does training old + new data using initial weights trained from old data decrease the training time appreciably? Are there any other practical considerations for dealing with this type of situation?
To narrow it down, the particular models I am looking at tend to be complicated ones for predictive (not explanatory) purposes. Neural networks, word-vectors etc. I am not using any form of regularisation.
