I have a neural network with 3 hidden layers and I'm unsure about the number of hidden nodes for each layer.

Should the number of hidden nodes stay constant between the hidden layers, e.g. 500 nodes for each layer, or should it decrease or decrease?

Is there some rule about how to choose the number of nodes per hidden layer?

up vote 2 down vote accepted

There's an excellent writeup to this question (and to the question of 'how many hidden layers?' as well) at https://stackoverflow.com/questions/10565868/what-is-the-criteria-for-choosing-number-of-hidden-layers-and-nodes-in-hidden-la . It may be disappointing to find that there are few hard-and-fast rules, and if there are, they are often mathematically or logically suspect. Also, another answer in that thread referenced this webpage: ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu .

Alternatively, depending on how computationally intensive it is to train your network, you can use various optimization algorithms to try to find it.

As for the more general question of whether or not layer size should stay constant, I would suggest considering that as a dimensionality-reduction procedure. Would you want your data to be compressed into a lower dimensional form and lose some information? This can be a positive or negative thing. For image compression, it's a requirement. See http://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/Applications/imagecompression.html for references on 'bottleneck' layers with image compression.

The type of problem which I would want to have big -> small -> big or some variety of that would probably involve a high dimensional source of data which I would like to compress and then learn features from. If you think that this describes your problem, then perhaps it is a valid approach to use more hidden units, feed into fewer units, then expand the layer out again.

  • Thanks for your answer. I have a dataset of fMRI images (brain scans from mice) which is divided into 4 classes (different drug doses applied). There are 37 images (data points) and the feature size is around 1000. I want to use the Neural Network for classifiaction. Would you use in this case the same number of nodes in each hidden layer? – machinery Jul 10 '15 at 22:46
  • I would partition the data up into training/test/validation and then try it with a bunch of different parameter settings; there probably isn't a short and sweet answer with a nice formulaic justification. Perhaps start out by looking at network sizes which are of similar size as your data's dimensionality and then vary the size of the hidden layers by dividing by 2 or multiplying by 2 and so on. If you have 3 hidden layers, you're going to have n^3 parameter configurations to check if you want to check n settings for each layer, but I think this should still be feasible. – Christopher Krapu Jul 10 '15 at 23:03
  • Ran into the character limit on the last one. Answering your question requires some careful examination of the image files (how noisy are they? are the features vaguely linear? Is there lots of structure in each image?) which is good practice for getting a feel for the effectiveness of various network designs. – Christopher Krapu Jul 10 '15 at 23:09
  • Thanks again for the answer. Just as a side note, how much epochs would you recommend? Second, currently I only have training and testing splits (leave-one-out cross-validatoin). That means I loop over all parameter settings doing a cross-validation. Will this introduce a large bias? Using training/test/validation splits requires even more runtime and I have less data points. Moreover, in the end I need to now the optimal parameter values. – machinery Jul 10 '15 at 23:36
  • Regarding your questions about the image files: The images are pretty noisy (although smoothed by a filter). The features are more or less linear. I don't know what you mean with structure in image. – machinery Jul 10 '15 at 23:38

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.