7
$\begingroup$

There's an old rule of thumb for multivariate statistics that recommends a minimum of 10 cases for each independent variable. But that's often where there is one parameter to fit for each variable.

Why I'm asking: I'm working through a textbook example that uses 500 Training cases (out of 25000 in the data set) with 15 predictor variables and one hidden layer with 8 hidden nodes. So we're estimating 153 weights. Of the 500 cases, there are 129 1's and the rest 0's. so there are more weights than positive cases to be predicted. This seems wrong. The resulting model overfits (but validation is not covered in this textbook problem).

So, what's a guide to the minimum? 10 times input variables? 10 times parameters to estimate? Something else?


There are related answers, but they seem to refer more to desirable sample sizes than minimum, e.g. How to get the data set size required for neural network training?

Tradeoff batch size vs. number of iterations to train a neural network

or be unanswered Minimum training sample size required for a classifier

But, of course, I may have missed some good previous answer.

$\endgroup$
  • 1
    $\begingroup$ This is an interesting question that is a few months old. If you have found a clear answers to your question please consider answering your question. $\endgroup$ – NULL Aug 1 '17 at 11:49
  • $\begingroup$ An emprical estimate for minimum number of input samples among NN researchers is that we'd need at least number of parameters squared samples $\endgroup$ – NULL Aug 1 '17 at 11:53
  • $\begingroup$ Haven't found a clear answer yet. $\endgroup$ – zbicyclist Aug 2 '17 at 13:50
4
$\begingroup$

This is impossible to answer in general. If you're working on a problem with strongly predictive features, your task is easier -- smaller sample sizes will estimate a highly performant model. But a problem with only weakly-relevant features will struggle to find signal.

In extrema, if all of your features are pure noise, no network will generalize well, even if you have arbitrarily large volumes of data.

Clever regularization and feature selection can help. And if regularization and feature selection can change the number of parameters you need to estimate a network with a specific level of performance, then this question seems even more complicated than a simple guideline.

$\endgroup$
1
$\begingroup$

So, what's a guide to the minimum? 10 times input variables? 10 times parameters to estimate?

I read classic statistical advice to use the number of samples at least 10 times more than the number of parameters. This is vague, of course. If the problem is too noisy, you can demand 100 times more, or 1000 times more.

As mentioned by @Sycorax, there is no clear concise answer, but you can at least keep in mind that having the number of samples equal to the number of parameters will allow to draw a hyperplane (in your case) that divides perfectly your classes.

Take a look at the Vapnik-Chervonenkis dimension concept that can help articulate the problem more formally: VC dimension

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.