Do Neural Networks Always Need all 3 Initiating Rules for Neurons in Hidden Layers

Question

Did we always need the 3 rules quoted below in the end result for a neural network? Could 2 or just 1 make a formidable network? I am seeing that if all 3 rules were met by a network, the number of inputs > the number of neurons in the hidden layers > the number of outputs

3 initiating rules:

1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.

2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.

3. The number of hidden neurons should be less than twice the size of the input layer.

The 3 rules above are taken from an Introduction to Neural Networks for Java (second edition) by Jeff Heaton available as a Google Book.

Answer 1

No, these are just rules of thumb (dictionary definition: not intended to be strictly accurate or reliable for every situation) for designing a neural network. Quoting from the neural-nets FAQ,

These rules of thumb are nonsense because they ignore the number of training cases, the amount of noise in the targets, and the complexity of the function. Even if you restrict consideration to minimizing training error on data with lots of training cases and no noise, it is easy to construct counterexamples that disprove these rules of thumb.

You must experiment yourself with different number of hidden layers, and number of neurons in them pertaining to your problem statement. (A starting guide)

Take for example, autoencoders. A type of autoencoder, called undercomplete autoencoder, uses fewer hidden neurons compared to the input neurons. The sparse autoencoder include more hidden neurons than inputs, but only a small number of the hidden neurons are allowed to be active at once.