Did we always need the 3 rules quoted below in the end result for a neural network? Could 2 or just 1 make a formidable network? I am seeing that if all 3 rules were met by a network, the number of inputs > the number of neurons in the hidden layers > the number of outputs

3 initiating rules:

  1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.

  2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.

  3. The number of hidden neurons should be less than twice the size of the input layer.

The 3 rules above are taken from an Introduction to Neural Networks for Java (second edition) by Jeff Heaton available as a Google Book.

  • $\begingroup$ This is very outdated advice, there’s a million and one different neural networks architectures nowadays and I doubt any of them even thought about these rules. Best thing is just to experiment yourself though with keras or something and see what happens $\endgroup$
    – XanderJC
    Commented Jan 2, 2020 at 11:07

1 Answer 1


No, these are just rules of thumb (dictionary definition: not intended to be strictly accurate or reliable for every situation) for designing a neural network. Quoting from the neural-nets FAQ,

These rules of thumb are nonsense because they ignore the number of training cases, the amount of noise in the targets, and the complexity of the function. Even if you restrict consideration to minimizing training error on data with lots of training cases and no noise, it is easy to construct counterexamples that disprove these rules of thumb.

You must experiment yourself with different number of hidden layers, and number of neurons in them pertaining to your problem statement. (A starting guide)

Take for example, autoencoders. A type of autoencoder, called undercomplete autoencoder, uses fewer hidden neurons compared to the input neurons. The sparse autoencoder include more hidden neurons than inputs, but only a small number of the hidden neurons are allowed to be active at once.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.