Comparing Neural Networks

Question

Let's assume a multilayer perceptron with $l$ layers and $n_i$ neurons at each layer $i=1, \cdots, l$. The number of input neurons $n_1$ and the number of output neurons $n_l$ are fixed. Now I would like to compare different network architectures with each other under the following constraint:

• the number of connections between all neurons is constant or
• the number of neurons in the network is constant.

I was told to keep the number of neurons constant. But under this constraint I can maximize the number of connections between neurons by keeping only one hidden layer which results in a network of higher capacity (much more weights $w$).

In my opinion the number of connections, and thus the weights should be kept constant while playing around with the network's architecture.

I would like to know which constraint makes more sense?

Answer 1

I think the more general way to look at this is not in terms of "connections," which can be challenging to apply in the case of networks that are not multi-layer perceptrons, but instead in terms of parameters (weights and biases).

For example, there is a dramatic difference in the number of parameters in a GRU and LSTM cell. Keeping the number of cells the same implies that the LSTM network has many more parameters than the GRU network, and hence a larger capacity to learn.

Answer 2

It depends on what you want to compare, if you want to compares architectures under constraints, you can change between number of hidden layers or number of neurons per layer.

Number of connections between layers are constant depending on number of neurons.