4
$\begingroup$

Is there a way to estimate the amount of data (or the number of records) required to build a statistical model? I read few blogs and I feel that most of the responses concur that there is no way or it is extremely hard to predict sample size for an application.

One one blog asks for 10 times the total number of features, less if I am using regularized version of the ML model. A sample size of 50 seem to be the minimum.

One method suggests to build the model and check the generalization error. If the error is unacceptable, then gather more data and iterate.

Considering the data collection for my application to be time consuming and extremely costly, what options do I have in time-constrained business organization?

$\endgroup$
1
  • $\begingroup$ This seems like a sample-size problem. I tried to add that tag, but it didn't take. Maybe you want to try? $\endgroup$
    – Russ Lenth
    Aug 26, 2014 at 1:53

2 Answers 2

5
$\begingroup$

It is my understanding that random sampling is a mandatory condition for making any generalization statements. IMHO, other parameters, such as sample size, just affect probability level (confidence) of generalization. Furthermore, I believe that you have to calculate needed sample size, based on desired values of confidence interval, effect size, statistical power and number of predictors (this is based on Cohen's work - see References section at the following link). For multiple regression, you can use the following calculator: http://www.danielsoper.com/statcalc3/calc.aspx?id=1.

More information on how to select, calculate and interpret effect sizes can be found in the following nice and comprehensive paper, which is freely available: http://jpepsy.oxfordjournals.org/content/34/9/917.full.

If you're using R (and even, if you don't), you may find the following Web page on confidence intervals and R interesting and useful: http://osc.centerforopenscience.org/static/CIs_in_r.html.

Finally, the following comprehensive guide to survey sampling can be helpful, even if you're not using survey research designs. In my opinion, it contains a wealth of useful information on sampling methods, sampling size determination (including calculator) and much more: http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm.

$\endgroup$
2
$\begingroup$

I don't think there is any rule of thumb. It depends on the type of model you want to use (e.g., simple logistic regression vs. fancy neural net), your data (e.g., i.i.d. vs. highly dependent), the complexity of the relationship you're trying to model (e.g., no interactions vs. lots of interactions), and probably some other things I couldn't think of. The more complicated your problem is, the more data you'll need.

The "build model and check generalization error" approach is an excellent idea (I assume you're doing prediction). Note that the amount of data you need depends your algorithm or desired accuracy. For example, if you have 20 observations, a linear regression might work fine but a neural net definitely won't.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.