I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the available data. The problem is that my data load time is larger than a single epoch and I would hate to incur that data load cost for each epoch.
The alternative approach that yields some value is to load as much data as possible, train the model for a few hundred epochs, then load the next portion of the data and reiterate for the same amount of epochs. And repeat this until all my data is "seen" by the model.
Some of the specifics of the task at hand: The whole data set is about 200K samples. 20K of those fit in memory. An epoch on a 1080ti GPU take about 60s and the data load of 20K samples is about 120s. My model is relatively simple CNN for now although it will get more complex with time hence the desire to avoid incurring the data load time on every epoch.
Intuitively I understand that the multiple retrainings approach is sub-optimal as the model will tend to optimize for the latest portion of the data and "forget" the previous data but I would like 1) a more in-depth explanation of the downsides of that method and 2) if there are any ways to overcome them.
