Let's assume a multilayer perceptron with $l$ layers and $n_i$ neurons at each layer $i=1, \cdots, l$. The number of input neurons $n_1$ and the number of output neurons $n_l$ are fixed. Now I would like to compare different network architectures with each other under the following constraint:
- the number of connections between all neurons is constant or
- the number of neurons in the network is constant.
I was told to keep the number of neurons constant. But under this constraint I can maximize the number of connections between neurons by keeping only one hidden layer which results in a network of higher capacity (much more weights $w$).
In my opinion the number of connections, and thus the weights should be kept constant while playing around with the network's architecture.
I would like to know which constraint makes more sense?