2
$\begingroup$

Is there a way of estimating in advance the number of observations one needs to achieve a maximum prediction error given a machine learning problem?

There are a huge number of questions on CV and elsewhere relating to choosing sample size for estimation or hypothesis testing. I haven't seen, though, much discussion on how sample size relates to the prediction accuracy of a given model (SVM, logistic regression, etc).

Note that I'm not referring to splitting a data set into folds, but rather determining the size of the total data set itself.

My hunch is that this is a very contextual problem (for starters, if it were simple the answer would be much easier to find!). I imagine one would need very specific assumptions about the distributions of every feature and the ground truth, and then for a specific model with fixed hyperparameters you could derive the expected error. But, since one has so many models and hyperparameters to consider, is this a practical exercise? The alternative of collecting as much data as you can afford is much simpler, but it would be nice to know in advance if you can even reach the minimum for your application.

This relates to a previous question that is unanswered as of now.

$\endgroup$
1
$\begingroup$

you might find this article interesting: https://www.ncbi.nlm.nih.gov/pubmed/15572470.

It is about: Optimal number of features as a function of sample size for various classification rules. It includes some 3D-plots of the error rate, sample size and feature size, for a number of algorithms e.g. SVM.

Their research concluded that: First, the behavior of the optimal-feature-size relative to the sample size depends strongly on the classifier and the feature-label distribution. An immediate corollary is that one should be wary of rules-ofthumb generalized from specific cases. Second, the performance of a designed classifier can be greatly influenced by the number of features and therefore one should attempt to use a number close to the optimal number.

$\endgroup$
1
  • $\begingroup$ This is very useful, thanks for sharing! Seems from the paper like my intuition was more or less correct, and that some models are a bit more consistent between problems. $\endgroup$ – RedPanda Sep 17 '19 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.