Is there a way to incorporate new data into an already trained neural network without retraining on all my data in Keras?

Question

I have already trained a neural network on my data. In the future, I will receive some more data. How can I incorporate this data into my model without rebuilding it from scratch?

Answer 1

In keras, you can save your model using model.save and then load that model using model.load. If you call .fit again on the model that you've loaded, it will continue training from the save point and will not restart from scratch. Each time you call .fit, keras will continue training on the model. .fit does not reset model weights.

I would like to point out one issue that might arise from training your model this way though, and that issue is catastrophic forgetting. If you feed your model examples that differ significantly from previous training examples, it might be prone to catastrophic forgetting. This is basically when the neural network learns your new examples well and forgets all the previous examples because you are no longer feeding those examples to it. It arises because as optimizers get more efficient, the neural network will be get more efficient at fitting new data quickly - and the best way to fit the new data quickly might be to forget old data. If your future data is very similar to your current data, then this won't be a problem. But imagine you trained a named entity recognition system to recognize organizations. If in the future you feed it a bunch of data that teach it how to recognize people names, it might catastrophically forget how to recognize organizations.

Answer 2

The model will continue to learn each time you call model.fit.

You can save a model using model.save and load it with load_model.

Here is a simple example:

from keras.layers import SimpleRNN, TimeDistributed

model=Sequential()
model.add(SimpleRNN(input_shape=(None, 2),
                    return_sequences=True, 
                    units=5))
model.add(TimeDistributed(Dense(activation='sigmoid', units=3)))
model.compile(loss = 'mse', optimizer = 'rmsprop')
model.fit(inputs, outputs, epochs = 500, batch_size = 32)

model.save('my_model.h5')

from keras.models import load_model
model = load_model('my_model.h5')

# continue fitting
model.fit(inputs, outputs, epochs = 500, batch_size = 32)