# Confidence interval for classification error: binomial assumption vs. bootsrap resampling

I am developing a classifier using a set of N patterns, where N~1000. I am using K-fold cross-validation (with K=5) and computing the probability of classification error p (typical value is p=0.03). I am also computing estimates of the 95% confidence interval; I do this by assuming the classifier output is binomial distributed (see Brown, Cai, & DasGupta, Anirban, “Interval Estimation for a Binomial Proportion”, Statistical Science, 16, 2, 2001, pp. 101-133).

Now to my question. I have also heard mention of using bootstrapping to measure confidence intervals. I don't have a feel for why you would need to use a resampling method; can we not assume the number of errors to be binomial distributed? Does the cross-validation screw this up? Is resampling only necessary if N and p are such that directly estimating the confidence interval becomes difficult/impossible?

This seems like a question that would come up any time the performance of a classifier is being analyzed.

require(rms)
s <- f$stats c(cindex=s['C'], lr=s['Model L.R.'], prop=mean((predict(f, type='fitted') > cutoff)==y)) }  } h <- function(cutoff=0.5) { age <- rnorm(n, 50, 12) sex <- factor(sample(c('m','f'), n, replace=TRUE)) L <- (age-50)*.04 + .75*(sex=='m') y <- ifelse(runif(n) < plogis(L), 1, 0) if(is.na(cutoff)) cutoff <- mean(y) w <- rbind(ac(NULL, y, cutoff), ac(lrm(y ~ age), y, cutoff), ac(lrm(y ~ sex), y, cutoff), ac(lrm(y ~ age + sex), y, cutoff)) dimnames(w) <- list(c('none','age','sex','age+sex'), c('C-index','LR Chi-square','Proportion Correct')) print(round(w,3)) invisible(data.frame(age, sex, y)) } set.seed(26) h() C-index LR Chi-square Proportion Correct none 0.500 0.000 0.572 age 0.592 10.520 0.622 sex 0.589 12.373 0.588 age+sex 0.639 22.848 0.600  You can see that with no predictors you get 0.57 correct by predicting every outcome to be Y=1 (this is the observed proportion of Y=1). The sex variable adds highly significant information to the model as judged by the gold standard likelihood ratio$\chi^2$statistic, but adding sex to age actually decreases the proportion classified correctly, implying that the sex variable contains negative information. • Thanks for the input, but I'm still confused (I admit I don't have a lot of experience with all this). If I run a classifier on 1000 patterns and it chooses the correct class 750 times, then the estimated probability of correct classification is 0.75. I then assume a binomial distribution (with mean 750) for the number of correct decisions. So I compute the confidence interval based on this assumed binomial distribution (and if I'm not mistaken, others have commonly used this approach). My question: Is this approach valid, or is a more sophisticated approach such as bootstrapping needed? – rdp Apr 29, 2013 at 19:05 • You missed my point. The proportion classified correct is in many senses an invalid measure. Consider a proper accuracy score such as the Brier score or a semi-proper one such as c-index (ROC area). These measures play the predicted risk against the observed outcome. Apr 29, 2013 at 21:49 • Interesting; I'll have to read up on proper measures such as Brier score. Do you by chance have any recommended readings to get me up to speed on this? – rdp Apr 30, 2013 at 14:33 • There are many primary papers on the subject. You can start with my course notes at biostat.mc.vanderbilt.edu/CourseBios330 Apr 30, 2013 at 19:04 • It is inappropriate to game the system by playing with the cutoff. Cutoffs need to be chosen on the basis of the cost/harm of false positives and false negatives [and even then a cutoff only works if costs are constant over all subjects]. For this particular case if you were to use the cutoff of 0.57 you find that the age variable has negative information because a model with age has a proportion correct of 0.56 even though it has a$\chi^2\$ of 10.5 when added to the null model. For most projects the utility/loss/cost function (necessary for classification to really work) is unavailable. May 3, 2013 at 20:09