Should the model be trained in every Bootstrap iteration? I am working in a classification scenario and I want to calculate a 95% confidence interval for my test set error rate. 
When I am using the Bootstrapping method:


*

*Shall I train the model once on the training dataset, and resample the testing data set N times to obtain N estimates of the error?

*Or shall I be, training the model on a resampled training dataset for every bootstrap iteration and estimating the error on a resampled testing dataset under that model iteration?


I feel like it's the second but I am having a discussion with a friend and we disagree...
 A: I guess you refer to the bootstrap estimators of an error rate as suggested in

B. Efron: Estimating the Error Rate of a Prediction Rule: Improvement on
  Cross-Validation. Journal of the American Statistical Association 78, pp. 316-331 (1983)

The article seems to be inaccessible behind a paywall, but Andrew Webb ("Statistical Pattern Recognition", 2nd ed., 2002) describes the algorithm for the bootstrap zero estimator as follows:
Input: training set T = {x_1, ..., x_n}  
Output: bootstrap estimator ê_boot of error rate  
For k = 1, ..., K  
    Draw from T randomly n traning samples S_k = { y_1, ..., y_n } with replacemment
    Train Classifier with S_k
    ê_A <- error rate with S_k as test set
    ê_T <- error rate with T as test set
    B_k <- ê_T - ê_A
EndFor
Train Classifier with T
ê_A <- error rate with T as test set
ê_boot <- ê_A + \frac{1}{K}\sum_{k=1}^{K} B_k
Return ê_boot

Obviously this estimator requires to train the classifier K+1 times. It is not sufficient to only sample the test data. The same applies to Efron's .632 bootstrap estimator, about which is said:

The .632 estimator performed best in the sampling experiments but has the weakest theoretical justification. (Hand, 1986)

