How to perform Neural Network modelling effectively?

Question

What should be the ratio of number of observations and number of variables? How to detect overfitting in the neural network model and what are the ways to avoid overfitting? If I want to perform classification with Neural Network, should the classes have equal frequency? Please help me out.

Answer 1

The advice I would give is as follows:

(i) Exhaust the possibilities of linear models (e.g. logistic regression) before going on to neural nets, especially if you have many features and not too many observations. For many problems a Neural Net does not out-perform simple linear classifiers, and the only way to find out if your problem is in this category is to try it and see.

(ii) Investigate kernel methods (e.g. SVM, kernel logistic regression), Gaussian process models first. In both cases over-fitting is effectively controlled by tuning a small number of hyper-parameters. For kernel methods this is often performed by cross-validation, for Gaussian process models this is performed by maximising the marginal likelihood (also known as the Bayesian "evidence" for the model). I have found it is much easier to get a reasonable model using these methods than with neural networks, as the means of avoiding over-fitting is so much more straightforward.

(iii) If you really want to use a neural network, start with a (regularised) radial basis function network, rather than a feedforward MLP type network.

(iv) If yo do use an MLP, then use regularisation. If you do, it will be less sensitive to choices about architecture, such as optimising the number of hidden units. Instead, all you have to do is choose a good value for the regularisation parameter. MacKay's Bayesian "evidence framework" provides a good method for setting the regularisation parameter. If you use regularisation, then the number of observations and number of variables becomes much less of an issue.

To detect over-fitting, simply perform cross-validation to test generalisation performance.

As for classes having equal frequencies, the thing to remember is that if you train a model with a balanced training set, but the classes are not balanced in the operational data, then the model is very likely to under-predict the minority class. If you use a probabilistic classifier such as logistic regression or a neural network, you can always correct the estimated probabilities to account for that after training. If your dataset is very imbalanced, I would recommend differential weighting of patterns from the positive and negative classes, with the weighting factors selected by cross-validation.

However, when the classes are very unbalanced, it is normally the case that false-negative and false-positive errors have difference costs (e.g. in medical screening tests a false-negative is much worse than a false-positive). So often all you need to do is include the misclassification costs into the error function used to train the network.

If you are a MATLAB user (like me) I can strongly recommend the NETLAB software (Ian Nabney and Chris Bishop) or the software that goes with the book Gaussian Process for Machine Learning by Rasmussen and Williams. I can else strongly recommend the book "Neural networks for pattern recognition" by Chris Bishop for anyone starting out in neural nets. It is a brilliant book, and covers the material with great clarity and the minimum level of maths required to really understand what you are doing, and most of it is implemented in the NETLAB software (which may also run under Octave).

HTH

P.S. The best way of modelling with a neural net is probably to use a Bayesian approach based on Hybrid Monte Carlo (HMC), as developed by Radford Neal. In general problems start in modelling when you try and optimise some parameters and you end up over-fitting. The best solution is to never optimise anything and marginalise (integrate) over parameters instead. Sadly this integration can't be performed analytically, so you need to use sampling based approaches instead. However, this is (a) computationally expensive and (b) a bit of a "black art" and required deep understanding and experience.