3
$\begingroup$

I am using a support vector machine for binary classification on a sample of size 150 (75 of each class). I am using 5-fold stratified cross-validation to estimate the expected prediction error, i.e. the expected generalization error for a random (stratified) training set of size 120. I am using 0-1 loss for the prediction error. My estimate for the expected prediction error from the cross-validation is unbiased, however since the training and test sets are not independent I don't see how to calculate confidence intervals (standard errors). However I need confidence intervals in order to show that my good result is not just a peculiarity of my dataset. Can anyone give a good method to estimate confidence intervals?

$\endgroup$
2
$\begingroup$

My estimate for the expected prediction error from the cross-validation is unbiased, however since the training and test sets are not independent

If training and test set are not independent, how can you be sure the estimate is not biased?

Assuming with "not independent" you refer to the fact that between each pair of surrogate models of the cross validation $\frac{n (k - 2)}{k}$ of the training cases are shared:

I need confidence intervals in order to show that my good result is not just a peculiarity of my dataset

If I understand you correctly, this is not possible if you want to generalize to other data sets of the same type and your modeling alogrithm. See Bengio, Y. and Grandvalet, Y.: No Unbiased Estimator of the Variance of K-Fold Cross-Validation Journal of Machine Learning Research, 2004, 5, 1089-1105

In contrast, if you want to generalize to prediction of unknown cases by the particular model you trained on that particular data set (i.e. the "peculiarities" of the training data should be included, just not of those of the test cases), binomial confidence intervals with $n$ = total number of cases account for the uncertainty due to testing only $n$ cases. This assumes your model (and surrogate models) are stable, otherwise you'd need to account for the instability variance.


Update: where does that difference come from / what variance is missing wrt. the Bengio question?

The difference is that for estimating the performance of the training algorithm you need to know how much the estimated model or its predictions vary wrt. to the actual training set (model stability). In order to decompose these sources of variance, you'd need to measure performance for a number of disjunct training sets. Unfortunately, you typically don't have enough cases to look at the variance in predictions of disjunct training sets.

Iterated CV allows you to have a glimpse at this by measuring the stability while not with respect to a completely new training set, but at least with respect to exchanging a few cases.

$\endgroup$
  • $\begingroup$ Thank you for you answer.With unbiased, I mean that the expectation of the cross-validation over all possible datasets of size 150 is the expected prediction error for a training set of size 120. Of course we can't converge to that value as long as we have only one dataset. The article you are citing says so as well: "While it is known that cross-validation provides an unbiased estimate of EPE, it is also known that its variance may be very large (Breiman, 1996)." $\endgroup$ – schubeda Aug 18 '15 at 12:32
  • $\begingroup$ As there is no unbiased estimator, does anyone know an estimator with small bias? $\endgroup$ – schubeda Aug 18 '15 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.