Regarding the size of training data for building classifier

Question

When we build a classifier, like SVM or Naive Bayesian, are there any generic rules or theoretical derivations on the size of training data set? For example, to train a SVM-based classifier, what should be the minimum size of training data in terms of feature space and some target performance metrics, such as precision and recall?

Answer 1

Just a rule of thumb: for the training dataset size, take 90% of your original dataset (the one for which you have labels / you know the real class of each sample).

The remaining 10% will be your test set.

I recommend AUC as the metric to compute on your test set. Look at the ROC curve also, since AUC is just a number.

It is better to randomize the order of your samples prior to cutting them in 90%, 10%.

Answer 2

There is no "minimum size" requirement. It is all about how variable our estimates for model performance are.

Especially for small(ish) sample sizes (<1000's) we should be looking at resampling techniques like bootstrap or repeated $k$-fold cross-validation to quantify the variability of our performance estimates. CV.SE already has some great threads on this matter; see for example: How to split the dataset for cross validation, learning curve, and final evaluation? and How large a training set is needed?.

Unless the available data are numerous and we have good reasons to believe a single hold-out test set would reflect the pattern observed on the training set, I would refrain from using a "single test-set". To give an anecdotal example of this: Kaggle competitions commonly withhold a third or even half of the available sample data for determining the final winner; that way a model's ability to generalise to "unseen" data is "properly rewarded".