How can a pretrained network specialize without "forgetting" the general principles it's learned when only limited datasets are available?
I am trying to train a set of neural networks to model pavement degredation. There are many different combinations, each of which behaves differently, and I am planning to train a different network for each. I have about 400 "good" samples showing the behavior I wish to model for each of the ~60 models.
I did a proof of concept by training a model with one of the larger datasets. It was very accurate for the data I had, but failed to generalize well - in some places the pavement was predicted to improve in quality (which doesn't happen in real life) simply because I didn't have sufficient data for that combination of values.
To solve this issue I decided to use a generic network as a starting place and then teach the ANNs to specialize for each specific model. The generic network works great - the curves all go in the right direction and it has plenty of data (25000+) to look at with good distribution.
When training with the smaller datasets, the network loses its ability to generalize - sometimes the degredations are predicted to go in reverse after a small amount of training, as small as 10 iterations. The training data does not include any bad data of that sort, but it does have a tendency to cluster around certain common combinations.
I'm looking for techniques that allow a pretrained network to specialize and not "unlearn" the general priciples it has picked up. If there are any improvements to architecture or to learning rates, etc. I'd be happy to hear them. This is also my first non-toy neural network, so any advice would be appreciated.
My network architecture is all sigmoid neurons, arranged in a 16x8x8x1 network. I'm using a Mean Squared Error cost function (makes sense for my data, and has given good results for the proof of concept). I'm using the synaptic.js library.