# Should the model be trained in every Bootstrap iteration?

I am working in a classification scenario and I want to calculate a 95% confidence interval for my test set error rate.

When I am using the Bootstrapping method:

• Shall I train the model once on the training dataset, and resample the testing data set N times to obtain N estimates of the error?
• Or shall I be, training the model on a resampled training dataset for every bootstrap iteration and estimating the error on a resampled testing dataset under that model iteration?

I feel like it's the second but I am having a discussion with a friend and we disagree...

I guess you refer to the bootstrap estimators of an error rate as suggested in

B. Efron: Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation. Journal of the American Statistical Association 78, pp. 316-331 (1983)

The article seems to be inaccessible behind a paywall, but Andrew Webb ("Statistical Pattern Recognition", 2nd ed., 2002) describes the algorithm for the bootstrap zero estimator as follows:

Input: training set T = {x_1, ..., x_n}
Output: bootstrap estimator ê_boot of error rate
For k = 1, ..., K
Draw from T randomly n traning samples S_k = { y_1, ..., y_n } with replacemment
Train Classifier with S_k
ê_A <- error rate with S_k as test set
ê_T <- error rate with T as test set
B_k <- ê_T - ê_A
EndFor
Train Classifier with T
ê_A <- error rate with T as test set
ê_boot <- ê_A + \frac{1}{K}\sum_{k=1}^{K} B_k
Return ê_boot


Obviously this estimator requires to train the classifier K+1 times. It is not sufficient to only sample the test data. The same applies to Efron's .632 bootstrap estimator, about which is said:

The .632 estimator performed best in the sampling experiments but has the weakest theoretical justification. (Hand, 1986)

• What will be the typical value of K? Sounds like this will be not practical for models that need a lot of time to train like neural networks or gradient boosting trees if K is 1000 like typical bootstrapping. – Louis Yang Mar 3 at 10:05
• @LouisYang You are right: if training takes days or weeks, this is not feasible. Note, however, that leave-one-out requires even more training operations, because usually the number of training samples is greater than the recommended value for K (for parametric bootstraps it is smaller than for non-parametric bootstraps). – cdalitz Mar 4 at 11:46