0
$\begingroup$

After having this exact same issue with caret, I arrived at this thread. However, I do not intuitively understand why this answer is correct.

Why can't there be more weights than the number of observations? Is this a bug/idiosyncrasy in this particular R package or is there a statistical reason for it?

$\endgroup$
3
$\begingroup$

It’s an issue with the particular software, maybe not a bug, but at least a matter of how the software performs, not neural networks themselves.

Consider training an MNIST network in Keras. You can have stellar out-of-sample accuracy when you have more than 60,000$^{\dagger}$ weights, so certainly a neural network model allows for more weights than observations.

$^{\dagger}$There are 60,000 training images in the MNIST data set.

$\endgroup$
2
$\begingroup$

Not only neural networks can have more weights, then samples, but there are some preliminary results showing that so called overparametrized neural networks (ones that have more parameters then samples) can overperform smaller networks. Below you can see figure by Belkin et al (2019) illustrate the phenomenon observed in some experiments, where the test error first falls with growing number of hidden units, then starts overfitting when the number of hidden units approaches the number of samples, but after surpassing the interpolation threshold (at this point the network is able to memorize the training dataset), but then it starts falling again with increasing complexity of the network.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.