2
$\begingroup$

Let's assume a multilayer perceptron with $l$ layers and $n_i$ neurons at each layer $i=1, \cdots, l$. The number of input neurons $n_1$ and the number of output neurons $n_l$ are fixed. Now I would like to compare different network architectures with each other under the following constraint:

  • the number of connections between all neurons is constant or
  • the number of neurons in the network is constant.

I was told to keep the number of neurons constant. But under this constraint I can maximize the number of connections between neurons by keeping only one hidden layer which results in a network of higher capacity (much more weights $w$).

In my opinion the number of connections, and thus the weights should be kept constant while playing around with the network's architecture.

I would like to know which constraint makes more sense?

$\endgroup$

2 Answers 2

2
$\begingroup$

I think the more general way to look at this is not in terms of "connections," which can be challenging to apply in the case of networks that are not multi-layer perceptrons, but instead in terms of parameters (weights and biases).

For example, there is a dramatic difference in the number of parameters in a GRU and LSTM cell. Keeping the number of cells the same implies that the LSTM network has many more parameters than the GRU network, and hence a larger capacity to learn.

$\endgroup$
2
  • $\begingroup$ In case of a multilayer perceptron network, what are its parameters? Only the weights and biases? $\endgroup$
    – Gilfoyle
    Aug 1, 2018 at 7:22
  • $\begingroup$ @Samuel That's correct. If you've found my answer helpful, please consider up-voting or accepting. Thanks. $\endgroup$
    – Sycorax
    Aug 11, 2018 at 17:24
0
$\begingroup$

It depends on what you want to compare, if you want to compares architectures under constraints, you can change between number of hidden layers or number of neurons per layer.

Number of connections between layers are constant depending on number of neurons.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.