I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the available data. The problem is that my data load time is larger than a single epoch and I would hate to incur that data load cost for each epoch.

The alternative approach that yields some value is to load as much data as possible, train the model for a few hundred epochs, then load the next portion of the data and reiterate for the same amount of epochs. And repeat this until all my data is "seen" by the model.

Some of the specifics of the task at hand: The whole data set is about 200K samples. 20K of those fit in memory. An epoch on a 1080ti GPU take about 60s and the data load of 20K samples is about 120s. My model is relatively simple CNN for now although it will get more complex with time hence the desire to avoid incurring the data load time on every epoch.

Intuitively I understand that the multiple retrainings approach is sub-optimal as the model will tend to optimize for the latest portion of the data and "forget" the previous data but I would like 1) a more in-depth explanation of the downsides of that method and 2) if there are any ways to overcome them.

  • $\begingroup$ Hello Георги, and welcome to Cross Validated. While I can suss the gist of your post and could probably lguess what question(s) you mean to ask, it would be best if you were to delineate them yourself. Exactly what would you like to know? The more focused the question, the higher the probability of a useful response. $\endgroup$ – Peter Leopold May 13 '19 at 15:31
  • $\begingroup$ I have two questions: 1) the downsides of using the retraining method. and 2) what are some ways to overcome those downsides. I am afraid I don't know more about the subject to be more specific. $\endgroup$ – Георги Кременлиев May 13 '19 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.