27
$\begingroup$

In text classification, I have a training set with about 800 samples, and a test set with about 150 samples. The test set has never been used, and waiting to be used until the end.

I am using the whole 800 sample training set, with 10 fold cross validate while tuning and tweaking classifiers and features. This means I do not have a separate validation set, but each run out of the 10 fold, a validation set is selected automatically.

After I will be satisfied with everything and want to enter the final stage of evaluation, I will train my classifiers on the whole 800 samples. And test on the 150 sample test set.

Is my understanding the such usage of cross validation in text classification correct? Is this practice valid?

Another question w.r.t. cross validation is:

instead of 10fold, I also tried leave one out as a general indicator for performance. Because for leave one out, it is not possible to have info about f1/precision/recall, I wonder what is the relationship between the accuracy from leave-one-out and metrics from 10fold?

Any insights would be highly appreciated.


Edit:

This is a quite nice introduction to cross-validation. It also refers to other research papers.

$\endgroup$
  • 3
    $\begingroup$ Leave-one-out estimators are unbiased, while 10fold cross-validation will tend to give you a biased (towards lower errors). However, the unbiasedness comes at the price of a high variance. $\endgroup$ – blubb Nov 24 '11 at 7:32
  • $\begingroup$ @Simon, I think It depends on a complexity of problem. Doesn't it? $\endgroup$ – Biostat Nov 24 '11 at 12:41
  • $\begingroup$ @blubb: LOO in certain situations can have a large pessimistic bias. Variance of LOO and a single run of 10-fold CV are usually very similar. Optimistic bias (too low error estimates) here does not come from the choice of resampling, but from the fact that thecross validation is used already for data driven optimization. After that, another independent validation is needed. That can be an "outer" loop of cross validation as well (without optimistic bias!) $\endgroup$ – cbeleites supports Monica Sep 27 '13 at 19:45
15
$\begingroup$

You have indeed correctly described the way to work with crossvalidation. In fact, you are 'lucky' to have a reasonable validation set at the end, because often, crossvalidation is used to optimize a model, but no "real" validation is done.

As @Simon Stelling said in his comment, crossvalidation will lead to lower estimated errors (which makes sense because you are constantly reusing the data), but fortunately this is the case for all models, so, barring catastrophy (i.e.: errors are only reduced slightly for a "bad" model, and more for "the good" model), selecting the model that performs best on a crossvalidated criterion, will typically also be the best "for real".

A method that is sometimes used to correct somewhat for the lower errors, especially if you are looking for parsimoneous models, is to select the smallest model/simplest method for which the crossvalidated error is within one SD from the (crossvalidated) optimum. As crossvalidation itself, this is a heuristic, so it should be used with some care (if this is an option: make a plot of your errors against your tuning parameters: this will give you some idea whether you have acceptable results)

Given the downward bias of the errors, it is important to not publish the errors or other performance measure from the crossvalidation without mentioning that these come from crossvalidation (although, truth be told: I have seen too many publications that don't mention that the performance measure was obtained from checking the performance on the original dataset either --- so mentioning crossvalidation actually makes your results worth more). For you, this will not be an issue, since you have a validation set.

A final warning: if your model fitting results in some close competitors, it is a good idea to look at their performances on your validation set afterwards, but do not base your final model selection on that: you can at best use this to soothe your conscience, but your "final" model must have been picked before you ever look at the validation set.

Wrt your second question: I believe Simon has given your all the answers you need in his comment, but to complete the picture: as often, it is the bias-variance trade-off that comes into play. If you know that, on average, you will reach the correct result (unbiasedness), the price is typically that each of your individual calculations may lie pretty far from it (high variance). In the old days, unbiasedness was the nec plus ultra, in current days, one has accepted at times a (small) bias (so you don't even know that the average of your calculations will result in the correct result), if it results in lower variance. Experience has shown that the balance is acceptable with 10-fold crossvalidation. For you, the bias would only be an issue for your model optimization, since you can estimate the criterion afterwards (unbiasedly) on the validation set. As such, there is little reason not to use crossvalidation.

$\endgroup$
  • $\begingroup$ "but your "final" model must have been picked before you ever look at the validation set." Nice. $\endgroup$ – Mooncrater Aug 18 '17 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.