0
$\begingroup$

How can a pretrained network specialize without "forgetting" the general principles it's learned when only limited datasets are available?

I am trying to train a set of neural networks to model pavement degredation. There are many different combinations, each of which behaves differently, and I am planning to train a different network for each. I have about 400 "good" samples showing the behavior I wish to model for each of the ~60 models.

I did a proof of concept by training a model with one of the larger datasets. It was very accurate for the data I had, but failed to generalize well - in some places the pavement was predicted to improve in quality (which doesn't happen in real life) simply because I didn't have sufficient data for that combination of values.

To solve this issue I decided to use a generic network as a starting place and then teach the ANNs to specialize for each specific model. The generic network works great - the curves all go in the right direction and it has plenty of data (25000+) to look at with good distribution.

When training with the smaller datasets, the network loses its ability to generalize - sometimes the degredations are predicted to go in reverse after a small amount of training, as small as 10 iterations. The training data does not include any bad data of that sort, but it does have a tendency to cluster around certain common combinations.

I'm looking for techniques that allow a pretrained network to specialize and not "unlearn" the general priciples it has picked up. If there are any improvements to architecture or to learning rates, etc. I'd be happy to hear them. This is also my first non-toy neural network, so any advice would be appreciated.

My network architecture is all sigmoid neurons, arranged in a 16x8x8x1 network. I'm using a Mean Squared Error cost function (makes sense for my data, and has given good results for the proof of concept). I'm using the synaptic.js library.

Improve this question
$\endgroup$
3
  • $\begingroup$ I just was poking around on Meta and if this would be better over on CS, I can put it there instead. $\endgroup$
    – Josiah
    Commented Aug 11, 2016 at 18:13
  • $\begingroup$ Instead of partitioning the problem into multiple networks, would it be possible to use a single neural net? That would increase the chances of encountering sufficient data for the combinations of values that are missing in others. Train all of the smaller datasets on a single net in order to compensate for the fact that none of them is sufficient on its own to solve the problem. I imagine this would take a lot of data scrubbing/normalization/other prep work etc. but it might do the trick. Is this possible? Just my initial impression... $\endgroup$
    – SQLServerSteve
    Commented Aug 11, 2016 at 19:44
  • 1
    $\begingroup$ Thanks @SQLServerSteve - it's a possibility. That was kind of the technique I was trying to use by training a general network and then teaching it to specialize. I may end up going that route. $\endgroup$
    – Josiah
    Commented Aug 12, 2016 at 11:31

1 Answer 1

Reset to default
0
$\begingroup$

If you have a large number of samples and a large number of networks to train, training a generic network using the full training set as a base can work well. The generic network is trained as usual. To teach it to specialize, use a very low learning rate (0.0001) and low number of iterations (<10000) to prevent overtraining or a loss of accuracy. Supervise the training very carefully using a testing set that is not part of the training data to ensure that accurate generalizations aren't being "unlearned".

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.