Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

What is the best way to select parameters for a binary neural network classifier? More specifically I have 265 features ranked according to Mutual Information Criterion. I have to determine the optimal number of inputs and the optimal number of hidden layer nodes for use in a multi-layer perceptron (MLP).

Note: The selecting of the optimal number of hidden nodes and optimal number of input features are not independent.

share|improve this question
add comment

1 Answer

Let's take a step back. Why would you want to select input features and why would you not take as many hidden nodes as you can?

One reason is speed. The more input features and hidden nodes you take, the longer training takes. I think that this is nowadays negligible due to fast computers. If this is the case for you, I can give you tips on how to speed up MLPs.

Another reason is to control overfitting. However, the neural network commmunity has come up with more efficient ways to control overfitting, where the most popular is weight decay. Let's say you optimise the loss $L$ with your neural net, then you will instead optimise $L + \lambda \sum_i w_i^2$ where the $w_i$ are all your weights (not biases) in your network. The $\lambda$ should be tuned to give best performance on a held out validation set.

Other ways of doing that is to use dropout or early stopping. Searching for the latter term here will give you good answers.

My advice is to use all input features, a big number of hidden nodes and use only weight decay. If you code everything yourself, use dropout instead of weight decay since it's much better.

share|improve this answer
I've performed a simulation study with a fairly large number of datasets, and have found that using a large number of hidden layer neurons and weight decay is not a reliable procedure (at least using Bayesian regularisation). Using regularisation and choosing the hidden layer size seems to be a better approach. –  Dikran Marsupial Nov 15 '12 at 16:33
Dikran, can you give more details? E.g. what kind of optimisation you used and so forth. It sounds interesting. –  bayerj Nov 15 '12 at 18:13
Thanks for your response, I will have to look in to this before responding further. I'm implementing my neural network in SAS, don’t know whether I will be able to change the the optimization function. I would also appreciate more detail if possible. Tips to speed up MLPs are always appreciated. –  entropy Nov 15 '12 at 18:55
@DikranMarsupial, if you could comment on the related question. –  entropy Nov 16 '12 at 10:31
@bayerj the form of optimisation shouldn't be critical, but IIRC it was conjugate gradient descent (I used NETLAB for the experiments). I am writing up the paper with a colleague at the moment, which will have all the details. I had originally thought that big networks and regularisation was a safe approach and I was surprised by the result. –  Dikran Marsupial Nov 16 '12 at 10:39
show 3 more comments

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.