2
$\begingroup$

Can a multilayer feed-forward network with n units per layer perform worse than a similar network with $<n$ units (e.g. same topology, just more units in each hidden layer)? I would imagine that it could not (assuming no local mimima problems), because if it did, then the training algorithm could just reduce some of the weights to zero, basically making the more complex network the equivalent of the less complex one. Is that correct, or am I missing something?

$\endgroup$
2
$\begingroup$

It is possible that the additional units are over-fitting the data. Formal analysis of neural networks is limited to broad statements because they're exceedingly difficult to manipulate analytically.

An experiment to test this for a particular dataset, is to perform nested k-fold cross-validation. Select a $n$ observations of the data to perform nested k-fold cross-validation. Each subfold is fixed on the number of hidden units, but the training parameters are tuned for learning rate, momentum, weight decay and (if you're using drop-out, then) drop-out probability.

Repeat this as a function of $n$. Plot the accuracy metric as a function of hidden units, for each $n$.

I would expect for small $n$, the argmax to be a smaller number of hidden units. For large $n$ it might approach a higher number of hidden units.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.