1
$\begingroup$

I've heard that as a rule of thumb, the number of samples needed for a machine learning algorithm to get accurate results is ten times the number of degrees of freedom. For example, classifying an 8x8 bitmap as being a digit between 0 to 9 would require (8 * 8) * 10 = 640 training examples for accuracy.

If this is correct, where is the origin of this rule of thumb (sources would be appreciated). If it is not correct, what is an accurate method of determining the minimum number of training examples needed for accurate predictions? (again, citations would be appreciated)

Many Thanks, Michael

$\endgroup$

migrated from stackoverflow.com May 27 '16 at 21:33

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ this is simply useless and false. The only reason for such reasoning would be assumption that your data is independent feature-wise, and follows normal distribution with noise near to 0. Furthermore, you assume that it is a binary classification. Then you end up with 5 per samples per class/feature pair to approximate a univariate gaussian, and with 0 noise this might be good enough to discriminate. Either way - this is extremely bad rule of thumb, do not ever trust it, seriously $\endgroup$ – lejlot May 22 '16 at 13:14
1
$\begingroup$

The origin is that you have to be worried about overfitting with generalized linear models when you have too many predictors per record (or per case in the most rare class in case of time-to-event, logistic regression or softmax-regression). An initial rule of thumb was 10 or more cases per record, but for some situations that may be too much of a requirement. The point is that below some point point estimates are substantially exagerrated, standard errors do not do what you hope etc. and that predictions might be less reliable as a result.

However, this rule of thumb primarily serves as a warning for when you may be entering dangerous territory with a (non-pentalized) GLM fit using maximum likelihood. To overcome that, you can of course use regularization and still GLMs (e.g. LASSO logistic regression, elastic net etc.) in such situations (especially if you are primarily after predictions), as long as you tune your hyperparameters sensibly (e.g. cross-validation).

Of course, there are plenty of other methods (such as Bayesian methods, gradient boosting, neural networks and so on) that all have their particular ways of achieving some form of regularization (e.g. priors, tuning the number of trees, dropout etc.), which - when used approrpiately - allow us to do useful things with sparse (relative to the amount of predictors) data.

$\endgroup$
0
$\begingroup$

The rule of thump is more like a joke and not a rule. Imagine these two cases: (1) digit recognition with hand written data (2) digit recognition based on machine printed digits using a handful of fonts. So what do you think? Clearly, you need more data to capture the variety of input data in the former case.

The only correct rule-of-thump is to test your classification method on your data to see how it performs. It totally depends on the complexity of your data and the power of your classifier.

$\endgroup$
  • $\begingroup$ Hi, friendly helpful tip for you: it is is "rule of thumb." I'll erase the comment . $\endgroup$ – Antoni Parellada May 27 '16 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.