I am developing a classifier using a set of N patterns, where N~1000. I am using K-fold cross-validation (with K=5) and computing the probability of classification error p (typical value is p=0.03). I am also computing estimates of the 95% confidence interval; I do this by assuming the classifier output is binomial distributed (see Brown, Cai, & DasGupta, Anirban, “Interval Estimation for a Binomial Proportion”, Statistical Science, 16, 2, 2001, pp. 101-133).

Now to my question. I have also heard mention of using bootstrapping to measure confidence intervals. I don't have a feel for why you would need to use a resampling method; can we not assume the number of errors to be binomial distributed? Does the cross-validation screw this up? Is resampling only necessary if N and p are such that directly estimating the confidence interval becomes difficult/impossible?

This seems like a question that would come up any time the performance of a classifier is being analyzed.


1 Answer 1


The proportion classified correctly is an improper scoring rule (i.e., it is optimized by a bogus model) and is a discontinuous function of the data. Therefore the methods used to estimate its statistical properties need to be more carefully chosen. For bootstrapping, the particular variant of the bootstrap you choose will matter, whereas for proper and semi-proper scoring rules (e.g., ROC area = c-index) the bootstrap variant matters very little.

Here is an R example showing one problem with proportion classified correctly. The cutoff chosen for a predicted Y=1 is 0.5.

n <- 400
## Function to compute ROC area (C-index), proportion classified correctly
## and LR chi-square
ac <- function(f, y, cutoff=0.5) {
  if(!length(f)) c(0.5, 0, mean(y)) else {
    s <- f$stats
    c(cindex=s['C'], lr=s['Model L.R.'],
      prop=mean((predict(f, type='fitted') > cutoff)==y))


h <- function(cutoff=0.5) {  
  age <- rnorm(n, 50, 12)
  sex <- factor(sample(c('m','f'), n, replace=TRUE))
  L <- (age-50)*.04 + .75*(sex=='m')
  y <- ifelse(runif(n) < plogis(L), 1, 0)
  if(is.na(cutoff)) cutoff <- mean(y)
  w <- rbind(ac(NULL, y, cutoff), ac(lrm(y ~ age), y, cutoff),
             ac(lrm(y ~ sex), y, cutoff),
             ac(lrm(y ~ age + sex), y, cutoff))
  dimnames(w) <- list(c('none','age','sex','age+sex'),
                      c('C-index','LR Chi-square','Proportion Correct'))
  invisible(data.frame(age, sex, y))


        C-index LR Chi-square Proportion Correct
none      0.500         0.000              0.572
age       0.592        10.520              0.622
sex       0.589        12.373              0.588
age+sex   0.639        22.848              0.600

You can see that with no predictors you get 0.57 correct by predicting every outcome to be Y=1 (this is the observed proportion of Y=1). The sex variable adds highly significant information to the model as judged by the gold standard likelihood ratio $\chi^2$ statistic, but adding sex to age actually decreases the proportion classified correctly, implying that the sex variable contains negative information.

  • $\begingroup$ Thanks for the input, but I'm still confused (I admit I don't have a lot of experience with all this). If I run a classifier on 1000 patterns and it chooses the correct class 750 times, then the estimated probability of correct classification is 0.75. I then assume a binomial distribution (with mean 750) for the number of correct decisions. So I compute the confidence interval based on this assumed binomial distribution (and if I'm not mistaken, others have commonly used this approach). My question: Is this approach valid, or is a more sophisticated approach such as bootstrapping needed? $\endgroup$
    – rdp
    Apr 29, 2013 at 19:05
  • $\begingroup$ You missed my point. The proportion classified correct is in many senses an invalid measure. Consider a proper accuracy score such as the Brier score or a semi-proper one such as c-index (ROC area). These measures play the predicted risk against the observed outcome. $\endgroup$ Apr 29, 2013 at 21:49
  • $\begingroup$ Interesting; I'll have to read up on proper measures such as Brier score. Do you by chance have any recommended readings to get me up to speed on this? $\endgroup$
    – rdp
    Apr 30, 2013 at 14:33
  • $\begingroup$ There are many primary papers on the subject. You can start with my course notes at biostat.mc.vanderbilt.edu/CourseBios330 $\endgroup$ Apr 30, 2013 at 19:04
  • 2
    $\begingroup$ It is inappropriate to game the system by playing with the cutoff. Cutoffs need to be chosen on the basis of the cost/harm of false positives and false negatives [and even then a cutoff only works if costs are constant over all subjects]. For this particular case if you were to use the cutoff of 0.57 you find that the age variable has negative information because a model with age has a proportion correct of 0.56 even though it has a $\chi^2$ of 10.5 when added to the null model. For most projects the utility/loss/cost function (necessary for classification to really work) is unavailable. $\endgroup$ May 3, 2013 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.