10
$\begingroup$

I have a neural network with 3 hidden layers and I'm unsure about the number of hidden nodes for each layer.

Should the number of hidden nodes stay constant between the hidden layers, e.g. 500 nodes for each layer, or should it decrease or decrease?

Is there some rule about how to choose the number of nodes per hidden layer?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

There's an excellent writeup to this question (and to the question of 'how many hidden layers?' as well) at https://stackoverflow.com/questions/10565868/what-is-the-criteria-for-choosing-number-of-hidden-layers-and-nodes-in-hidden-la . It may be disappointing to find that there are few hard-and-fast rules, and if there are, they are often mathematically or logically suspect. Also, another answer in that thread referenced this webpage: ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu .

Alternatively, depending on how computationally intensive it is to train your network, you can use various optimization algorithms to try to find it.

As for the more general question of whether or not layer size should stay constant, I would suggest considering that as a dimensionality-reduction procedure. Would you want your data to be compressed into a lower dimensional form and lose some information? This can be a positive or negative thing. For image compression, it's a requirement. See http://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/Applications/imagecompression.html for references on 'bottleneck' layers with image compression.

The type of problem which I would want to have big -> small -> big or some variety of that would probably involve a high dimensional source of data which I would like to compress and then learn features from. If you think that this describes your problem, then perhaps it is a valid approach to use more hidden units, feed into fewer units, then expand the layer out again.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your answer. I have a dataset of fMRI images (brain scans from mice) which is divided into 4 classes (different drug doses applied). There are 37 images (data points) and the feature size is around 1000. I want to use the Neural Network for classifiaction. Would you use in this case the same number of nodes in each hidden layer? $\endgroup$
    – machinery
    Jul 10, 2015 at 22:46
  • $\begingroup$ I would partition the data up into training/test/validation and then try it with a bunch of different parameter settings; there probably isn't a short and sweet answer with a nice formulaic justification. Perhaps start out by looking at network sizes which are of similar size as your data's dimensionality and then vary the size of the hidden layers by dividing by 2 or multiplying by 2 and so on. If you have 3 hidden layers, you're going to have n^3 parameter configurations to check if you want to check n settings for each layer, but I think this should still be feasible. $\endgroup$
    – Christopher Krapu
    Jul 10, 2015 at 23:03
  • $\begingroup$ Ran into the character limit on the last one. Answering your question requires some careful examination of the image files (how noisy are they? are the features vaguely linear? Is there lots of structure in each image?) which is good practice for getting a feel for the effectiveness of various network designs. $\endgroup$
    – Christopher Krapu
    Jul 10, 2015 at 23:09
  • $\begingroup$ Thanks again for the answer. Just as a side note, how much epochs would you recommend? Second, currently I only have training and testing splits (leave-one-out cross-validatoin). That means I loop over all parameter settings doing a cross-validation. Will this introduce a large bias? Using training/test/validation splits requires even more runtime and I have less data points. Moreover, in the end I need to now the optimal parameter values. $\endgroup$
    – machinery
    Jul 10, 2015 at 23:36
  • $\begingroup$ Regarding your questions about the image files: The images are pretty noisy (although smoothed by a filter). The features are more or less linear. I don't know what you mean with structure in image. $\endgroup$
    – machinery
    Jul 10, 2015 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.