1
$\begingroup$

My approach is to think about ANN like a common statistical model. 2000 data for 2000 parameter is clearly not enough. However, if we got 10k data points then the training result might start to become meaningful. Is it correct?

In my case, the output can be simplify to a simple 0,1 classification.

I've recent seen an empirical rule stating that a sample size of number of parameters squared is roughly needed to train a neural network. Are there any sources for this empirical rule?

$\endgroup$
1
  • $\begingroup$ Number of parameters is not enough to estimate the number of samples. Also saying that you are working with a binary classification is not enough. The number of samples depend on the complexity of the model that you want to fit. You can think in terms of statistical model. In that case, the sample size depends on the parameters of the model but also on the expected effect size. Any empirical rule is based on some assumptions missing here. $\endgroup$
    – user289381
    Jul 4 '20 at 1:19
5
$\begingroup$

There is no simple answer to such question. You can train neural network with one sample, you'd just overfit to it. Moreover, there are some recent results that in some cases neural networks with few orders of magnitude more parameters than samples can achieve better test set performance than smaller networks. Such rules of thumb don’t even work for much simpler models, e.g. you can fit regularized linear regression to dataset that has less samples than the model parameters. So it is certainly not true that you need the number of samples to be equal to number of parameters squared.

It would also depend on what is your data, for example if it was multiclass classification with ten classes, this makes only 1000 samples per class, on average. Is it many, or little? Say that you are classifying animals based on photos. If there are approximately 360 official dog breeds, and crossbread dogs of many different looks, then having 1000 images of dogs would not catch even a fraction of possible variability of different dogs.

Also keep in mind that neural networks flourish when trained on huge amounts of data, if that is not the case, then I'd start with trying different machine learning algorithms first.

$\endgroup$
4
  • $\begingroup$ So you are recommending me to have as much data as possible and then try to over-parameterize the model? $\endgroup$
    – High GPA
    Jul 3 '20 at 22:26
  • $\begingroup$ @HighGPA overparametrized models are just an example, rather not practical one, but showing the point that such rules of thumb do not apply. $\endgroup$
    – Tim
    Jul 3 '20 at 22:28
  • $\begingroup$ So you believe that the mechanism explained by Belkin et al (2019) does not generally hold; it only holds for a very limited set of examples? $\endgroup$
    – High GPA
    Jul 3 '20 at 22:29
  • $\begingroup$ @HighGPA I didn't say so. This was just an example. Overparametrized model would not be a practical solution, since you would need to use huge computational power to train on a small dataset (e.g. for 10k samples, 100M parameters). It is impractical to use in real life. The point is that there is no simple relation between number of samples and number of parameters. $\endgroup$
    – Tim
    Jul 3 '20 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.