4
$\begingroup$

I am working in a classification scenario and I want to calculate a 95% confidence interval for my test set error rate.

When I am using the Bootstrapping method:

  • Shall I train the model once on the training dataset, and resample the testing data set N times to obtain N estimates of the error?
  • Or shall I be, training the model on a resampled training dataset for every bootstrap iteration and estimating the error on a resampled testing dataset under that model iteration?

I feel like it's the second but I am having a discussion with a friend and we disagree...

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

I guess you refer to the bootstrap estimators of an error rate as suggested in

B. Efron: Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation. Journal of the American Statistical Association 78, pp. 316-331 (1983)

The article seems to be inaccessible behind a paywall, but Andrew Webb ("Statistical Pattern Recognition", 2nd ed., 2002) describes the algorithm for the bootstrap zero estimator as follows:

Input: training set T = {x_1, ..., x_n}  
Output: bootstrap estimator ê_boot of error rate  
For k = 1, ..., K  
    Draw from T randomly n traning samples S_k = { y_1, ..., y_n } with replacemment
    Train Classifier with S_k
    ê_A <- error rate with S_k as test set
    ê_T <- error rate with T as test set
    B_k <- ê_T - ê_A
EndFor
Train Classifier with T
ê_A <- error rate with T as test set
ê_boot <- ê_A + \frac{1}{K}\sum_{k=1}^{K} B_k
Return ê_boot

Obviously this estimator requires to train the classifier K+1 times. It is not sufficient to only sample the test data. The same applies to Efron's .632 bootstrap estimator, about which is said:

The .632 estimator performed best in the sampling experiments but has the weakest theoretical justification. (Hand, 1986)

Improve this answer
$\endgroup$
2
  • $\begingroup$ What will be the typical value of K? Sounds like this will be not practical for models that need a lot of time to train like neural networks or gradient boosting trees if K is 1000 like typical bootstrapping. $\endgroup$
    – Louis Yang
    Mar 3, 2021 at 10:05
  • 1
    $\begingroup$ @LouisYang You are right: if training takes days or weeks, this is not feasible. Note, however, that leave-one-out requires even more training operations, because usually the number of training samples is greater than the recommended value for K (for parametric bootstraps it is smaller than for non-parametric bootstraps). $\endgroup$
    – cdalitz
    Mar 4, 2021 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.