3
$\begingroup$

What is the best way to select parameters for a binary neural network classifier? More specifically I have 265 features ranked according to Mutual Information Criterion. I have to determine the optimal number of inputs and the optimal number of hidden layer nodes for use in a multi-layer perceptron (MLP).

Note: The selecting of the optimal number of hidden nodes and optimal number of input features are not independent.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Let's take a step back. Why would you want to select input features and why would you not take as many hidden nodes as you can?

One reason is speed. The more input features and hidden nodes you take, the longer training takes. I think that this is nowadays negligible due to fast computers. If this is the case for you, I can give you tips on how to speed up MLPs.

Another reason is to control overfitting. However, the neural network commmunity has come up with more efficient ways to control overfitting, where the most popular is weight decay. Let's say you optimise the loss $L$ with your neural net, then you will instead optimise $L + \lambda \sum_i w_i^2$ where the $w_i$ are all your weights (not biases) in your network. The $\lambda$ should be tuned to give best performance on a held out validation set.

Other ways of doing that is to use dropout or early stopping. Searching for the latter term here will give you good answers.

My advice is to use all input features, a big number of hidden nodes and use only weight decay. If you code everything yourself, use dropout instead of weight decay since it's much better.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ I've performed a simulation study with a fairly large number of datasets, and have found that using a large number of hidden layer neurons and weight decay is not a reliable procedure (at least using Bayesian regularisation). Using regularisation and choosing the hidden layer size seems to be a better approach. $\endgroup$
    – Dikran Marsupial
    Commented Nov 15, 2012 at 16:33
  • $\begingroup$ Dikran, can you give more details? E.g. what kind of optimisation you used and so forth. It sounds interesting. $\endgroup$
    – bayerj
    Commented Nov 15, 2012 at 18:13
  • $\begingroup$ Thanks for your response, I will have to look in to this before responding further. I'm implementing my neural network in SAS, don’t know whether I will be able to change the the optimization function. I would also appreciate more detail if possible. Tips to speed up MLPs are always appreciated. $\endgroup$
    – entropy
    Commented Nov 15, 2012 at 18:55
  • $\begingroup$ @DikranMarsupial, if you could comment on the related question. $\endgroup$
    – entropy
    Commented Nov 16, 2012 at 10:31
  • $\begingroup$ @bayerj the form of optimisation shouldn't be critical, but IIRC it was conjugate gradient descent (I used NETLAB for the experiments). I am writing up the paper with a colleague at the moment, which will have all the details. I had originally thought that big networks and regularisation was a safe approach and I was surprised by the result. $\endgroup$
    – Dikran Marsupial
    Commented Nov 16, 2012 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.