I am developing a classifier using a set of N patterns, where N~1000. I am using K-fold cross-validation (with K=5) and computing the probability of classification error p (typical value is p=0.03). I am also computing estimates of the 95% confidence interval; I do this by assuming the classifier output is binomial distributed (see Brown, Cai, & DasGupta, Anirban, “Interval Estimation for a Binomial Proportion”, Statistical Science, 16, 2, 2001, pp. 101-133).
Now to my question. I have also heard mention of using bootstrapping to measure confidence intervals. I don't have a feel for why you would need to use a resampling method; can we not assume the number of errors to be binomial distributed? Does the cross-validation screw this up? Is resampling only necessary if N and p are such that directly estimating the confidence interval becomes difficult/impossible?
This seems like a question that would come up any time the performance of a classifier is being analyzed.