5
$\begingroup$

When we build a classifier, like SVM or Naive Bayesian, are there any generic rules or theoretical derivations on the size of training data set? For example, to train a SVM-based classifier, what should be the minimum size of training data in terms of feature space and some target performance metrics, such as precision and recall?

$\endgroup$
0
$\begingroup$

Just a rule of thumb: for the training dataset size, take 90% of your original dataset (the one for which you have labels / you know the real class of each sample).

The remaining 10% will be your test set.

I recommend AUC as the metric to compute on your test set. Look at the ROC curve also, since AUC is just a number.

It is better to randomize the order of your samples prior to cutting them in 90%, 10%.

$\endgroup$
0
$\begingroup$

There is no "minimum size" requirement. It is all about how variable our estimates for model performance are.

Especially for small(ish) sample sizes (<1000's) we should be looking at resampling techniques like bootstrap or repeated $k$-fold cross-validation to quantify the variability of our performance estimates. CV.SE already has some great threads on this matter; see for example: How to split the dataset for cross validation, learning curve, and final evaluation? and How large a training set is needed?.

Unless the available data are numerous and we have good reasons to believe a single hold-out test set would reflect the pattern observed on the training set, I would refrain from using a "single test-set". To give an anecdotal example of this: Kaggle competitions commonly withhold a third or even half of the available sample data for determining the final winner; that way a model's ability to generalise to "unseen" data is "properly rewarded".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.