I am working in a classification scenario and I want to calculate a 95% confidence interval for my test set error rate.

When I am using the Bootstrapping method:

  • Shall I train the model once on the training dataset, and resample the testing data set N times to obtain N estimates of the error?
  • Or shall I be, training the model on a resampled training dataset for every bootstrap iteration and estimating the error on a resampled testing dataset under that model iteration?

I feel like it's the second but I am having a discussion with a friend and we disagree...


I guess you refer to the bootstrap estimators of an error rate as suggested in

B. Efron: Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation. Journal of the American Statistical Association 78, pp. 316-331 (1983)

The article seems to be inaccessible behind a paywall, but Andrew Webb ("Statistical Pattern Recognition", 2nd ed., 2002) describes the algorithm for the bootstrap zero estimator as follows:

Input: training set T = {x_1, ..., x_n}  
Output: bootstrap estimator ê_boot of error rate  
For k = 1, ..., K  
    Draw from T randomly n traning samples S_k = { y_1, ..., y_n } with replacemment
    Train Classifier with S_k
    ê_A <- error rate with S_k as test set
    ê_T <- error rate with T as test set
    B_k <- ê_T - ê_A
Train Classifier with T
ê_A <- error rate with T as test set
ê_boot <- ê_A + \frac{1}{K}\sum_{k=1}^{K} B_k
Return ê_boot

Obviously this estimator requires to train the classifier K+1 times. It is not sufficient to only sample the test data. The same applies to Efron's .632 bootstrap estimator, about which is said:

The .632 estimator performed best in the sampling experiments but has the weakest theoretical justification. (Hand, 1986)

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.