11
$\begingroup$

I was wondering if there under some circumstances is possible for ANN's to perform better if you prune away some connections on them as for example:

Constructing one ANN by taking two multi-layered ANN's A and B in parallel (same input and output nodes) adding a few "communication" connections between the hidden layers of A and B?

Could one get better generalization results?

Is this somehow used in practice or does one always just use multi-layered fully connected networks?

$\endgroup$
7
$\begingroup$

Yes it is possible. Some people have looked at this problem in detail. Here is an old paper about a method to do so: Optimal brain damage

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Why detaching the nodes is better than regularization? I thought, with regularization there's no need to prune the connections - the "unnecessary" ones will just get very small weights and that's it. $\endgroup$ – andreister Feb 9 '12 at 9:23
  • $\begingroup$ @andreister I don't think it is better than regularization. I think it is an (early) alternative to regularization. It is a very old paper, regularization became mainstream in ML in the mid to late nineties. $\endgroup$ – carlosdc Feb 9 '12 at 17:07
7
$\begingroup$

As a rule of thumb, small and/or sparse networks generalise better. You can let your training algorithm weed out unecessary connections within a fixed-size network by applying some form of weight decay, or you can apply an algorithm that aims to optimise network architecture/topology itself through removing unecessary inputs, hidden nodes or connections.

Have a look at these references for ideas and starting points for further research, or look into the use of evolutionary algorithms to design, prune and optimise architectures.

  1. Castellano, G., Fanelli, A.M. (2000) 'Variable selection using neural-network models', Neurcomputing (31)
  2. Ji C., Psaltis D. (1997) 'Network Synthesis through Data-Driven Growth and Decay', Neural Networks Vol. 10, No. 6, pp. 1133-1141
  3. Narasimha P.L. et al (2008) 'An integrated growing-pruning method for feedforward network training', Neurocomputing (71), pp. 2831-2847
  4. Schuster, A. (2008) 'Robust Artificial Neural Network Architectures', International Journal of Computational Intelligence (4:2), pp. 98-104
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I would really like to hear more about the "look into the use of evolutionary algorithms to design, prune and optimise architectures" part of the answer. Maybe I will ask a question about it! $\endgroup$ – Artem Kaznatcheev Feb 9 '12 at 7:22
6
$\begingroup$

In most cases if you remove unnecesary connections you'll get better network. It is easy to overtrain (overfit) the network --- in which case it will perform poorly on validation dataset.

Pruning unnecesary connections will most probably reduce o overtraining probability. Please see: http://en.wikipedia.org/wiki/Overfitting .

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

Yes It is possible. We can consider, connection between computational unites, number of hidden layers, unites per hidden layer etc as hyper-parameters. It possible to find-out optimal values for these parameters by conducting a series of experiments.

For example:

Your can divide your data set as follows: Training set 60% of data, Cross-validation 20% of data, Testing 20% of data,

Then train your NN by using training data set and tuning parameter by using cross-validation data set.

Finally you can use your testing data set for evaluate the performance of your NN.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.