Catastrophic forgetting: Retraining a trained neural network with small data

Question

I have a fully connected deep neural network with 7 hidden layers, which is trained with around 20000 simulated materials data. And we've got a very small measurement dataset (size<200) which reflects the actual behaviour of these materials.

I'd like to retrain the trained neural network with these measurement data such that the predictions would also reflect the actual behaviour rather than the simulations.

However, when I retrained the model with this small data, the predictions don't seem to be stable (which they were previously with 20000 data) and the graphs (say some material property vs electric field graph) are not smooth with some jumps here and there.

So I have 2 questions.

1. Is it even possible to get stabilized predictions, after retraining with small data?
2. Does it make sense to mix the simulated and measurement datasets and train again and hope an improvement towards the actual behaviour of the material? (For me it doesn't make sense considering the sizes of datasets)

I know that a neural network can forget previously learned features easily upon learning new information, called Catastrophic Forgetting. What role does this play in my scenario? But people still use trained NNs like VGGNet and train it with their own data and achieve good results.

Could someone suggest me a way how I can use this small measurement data to make a stable impact?

Answer 1

Does the retraining improve your training error? What is the test error? Unstable predictions make me think that you are over fitting to the measurement set.

Are you retraining the whole network with your measurement data? I think that is way too small data set to retrain it completely. If that is the case you might consider to retrain only the last layer or at most a few of the last layers. Another possibility is to take the use the values in the last hidden layer as the input to a basic ML model, such as linear regression.