# Minimum Training size for simple neural net

There's an old rule of thumb for multivariate statistics that recommends a minimum of 10 cases for each independent variable. But that's often where there is one parameter to fit for each variable.

Why I'm asking: I'm working through a textbook example that uses 500 Training cases (out of 25000 in the data set) with 15 predictor variables and one hidden layer with 8 hidden nodes. So we're estimating 153 weights. Of the 500 cases, there are 129 1's and the rest 0's. so there are more weights than positive cases to be predicted. This seems wrong. The resulting model overfits (but validation is not covered in this textbook problem).

So, what's a guide to the minimum? 10 times input variables? 10 times parameters to estimate? Something else?

There are related answers, but they seem to refer more to desirable sample sizes than minimum, e.g. How to get the data set size required for neural network training?

Tradeoff batch size vs. number of iterations to train a neural network

or be unanswered Minimum training sample size required for a classifier

But, of course, I may have missed some good previous answer.

– NULL
Aug 1, 2017 at 11:49
• An emprical estimate for minimum number of input samples among NN researchers is that we'd need at least number of parameters squared samples
– NULL
Aug 1, 2017 at 11:53
• Haven't found a clear answer yet. Aug 2, 2017 at 13:50
• @NULL # of parameters squared is an interesting rule! Any reference/resource regarding this rule? Jul 3, 2020 at 22:05
• Minimum viable dataset might be what you wanted. Jan 16, 2021 at 12:07

This is impossible to answer in general. If you're working on a problem with strongly predictive features, your task is easier -- smaller sample sizes will estimate a highly performant model. But a problem with only weakly-relevant features will struggle to find signal.

In extrema, if all of your features are pure noise, no network will generalize well, even if you have arbitrarily large volumes of data.

Clever regularization and feature selection can help. And if regularization and feature selection can change the number of parameters you need to estimate a network with a specific level of performance, then this question seems even more complicated than a simple guideline.

So, what's a guide to the minimum? 10 times input variables? 10 times parameters to estimate?

I read classic statistical advice to use the number of samples at least 10 times more than the number of parameters. This is vague, of course. If the problem is too noisy, you can demand 100 times more, or 1000 times more.

As mentioned by @Sycorax, there is no clear concise answer, but you can at least keep in mind that having the number of samples equal to the number of parameters will allow to draw a hyperplane (in your case) that divides perfectly your classes.

Take a look at the Vapnik-Chervonenkis dimension concept that can help articulate the problem more formally: VC dimension

• Does this mean that if I have 100 classes, I need at least 10 samples per class? Mar 20 at 7:06
• @ruslaniv, no, I would not say this. Calculate the number of weights in your model and multiply by ten. But it is also known that "over-parametrization" in neural networks works well, meaning you can have many more params than cases, and you do few epochs. Mar 23 at 12:36