The 0.632 bootstrapping rule by Efron in `Estimating the error rate of a prediction rule: improvement on cross-validation'. According to Tibshirani's overview in The Elements of Statistical Learning, calculating the leave-one-out bootstrap estimator (the component weighted by 0.632) requires computing:

$$ \hat{\text{Err}}^{(1)} = \frac{1}{N}\sum_{i=1}^{N} \frac{1}{|C^{-1}|} \sum_{b\in C^{-1}} L(y_i, \hat{f}^{*b}(x_i)) $$

where we have $N$ points in our dataset, $L(y_i, \hat{f}^{*b}(x_i))$ is the loss predicted fitting bootstrap sample $b$ to our model and predicting on the $i$th training pair, and $C^{-1}$ is the set of indices of bootstrap samples that do not contain observation $i$.

The examples given work with classification, so observations $y_i$ are always discrete. In the case of regression, observations are continuous and most of them will be unique, causing $\hat{\text{Err}}^{(1)}$ to become similar to a jackknife predictor. Is that okay, or does the bootstrap simply not apply to regression problems?

EDIT I was confused over how the equation above works. It runs out you make b bootstrap samples before using it. That means selecting b sets, each with N indices. Thus $|C^{-1}|$ for index $i$ is the number of those sets which do not contain $i$. Should work just fine for regression.

  • 1
    $\begingroup$ Actually it is a linear combination of the $e_0$ estimate (proposed by Chatterjee and Chatterjee) and the resubstitution estimate with weight 0.632 to $e_0$ and weight 0.368 to resubstitution. The $e_0$ is not the same as leave-one-out.. $\endgroup$ – Michael Chernick Feb 9 '18 at 4:17
  • 2
    $\begingroup$ Bootstrapping is done in two ways for regression. You can bootstrap residuals or bootstrap the vector of the observed Ys and the covariates. There is nothing in the literature that I know of that uses some analogue to 632. You can find out about bootstrapping in regression in the book by Efron and Tibshirani or my book Bootstrap Methoods: A Practitioner's Guide. Discriminant analysis methods are also given in my book along with simulations comparing various methods. $\endgroup$ – Michael Chernick Feb 9 '18 at 4:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.