Number of nodes in hidden layers of neural network I have a neural network with 3 hidden layers and I'm unsure about the number of hidden nodes for each layer.
Should the number of hidden nodes stay constant between the hidden layers, e.g. 500 nodes for each layer, or should it decrease or decrease? 
Is there some rule about how to choose the number of nodes per hidden layer?
 A: There's an excellent writeup to this question (and to the question of 'how many hidden layers?' as well) at https://stackoverflow.com/questions/10565868/what-is-the-criteria-for-choosing-number-of-hidden-layers-and-nodes-in-hidden-la . It may be disappointing to find that there are few hard-and-fast rules, and if there are, they are often mathematically or logically suspect. Also, another answer in that thread referenced this webpage: ftp://ftp.sas.com/pub/neural/FAQ3.html#A_hu .
Alternatively, depending on how computationally intensive it is to train your network, you can use various optimization algorithms to try to find it. 
As for the more general question of whether or not layer size should stay constant, I would suggest considering that as a dimensionality-reduction procedure. Would you want your data to be compressed into a lower dimensional form and lose some information? This can be a positive or negative thing. For image compression, it's a requirement. See http://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/Applications/imagecompression.html for references on 'bottleneck' layers with image compression.
The type of problem which I would want to have big -> small -> big or some variety of that would probably involve a high dimensional source of data which I would like to compress and then learn features from. If you think that this describes your problem, then perhaps it is a valid approach to use more hidden units, feed into fewer units, then expand the layer out again.
