I have a fully connected deep neural network with 7 hidden layers, which is trained with around 20000 simulated materials data. And we've got a very small measurement dataset (size<200) which reflects the actual behaviour of these materials.
I'd like to retrain the trained neural network with these measurement data such that the predictions would also reflect the actual behaviour rather than the simulations.
However, when I retrained the model with this small data, the predictions don't seem to be stable (which they were previously with 20000 data) and the graphs (say some material property vs electric field graph) are not smooth with some jumps here and there.
So I have 2 questions.
- Is it even possible to get stabilized predictions, after retraining with small data?
- Does it make sense to mix the simulated and measurement datasets and train again and hope an improvement towards the actual behaviour of the material? (For me it doesn't make sense considering the sizes of datasets)
I know that a neural network can forget previously learned features easily upon learning new information, called Catastrophic Forgetting. What role does this play in my scenario? But people still use trained NNs like VGGNet and train it with their own data and achieve good results.
Could someone suggest me a way how I can use this small measurement data to make a stable impact?