Bootstrap intervals for predictions, how to interpret it?

Question

I want to come up with a way to get how confident I am in my predictions. I am not using a Bayesian model so I was thinking a bootstrap confidence interval would be good:

I would re-sample my original training set, train on it and predict. And repeat that n number of times. I would then gather the n predictions and I could get some bootstrap intervals on my prediction.

The way that I interpret that would be: the larger the interval, the more sensitive my prediction is to the variation in the training data, which would in turn mean that I am fairly uncertain about my prediction. The smaller the interval would mean the opposite.

The questions are:

1) Is this a sensible thing to do to start with? 2) Is there any other interpretation that could be made? In this case, it is not true that for a 95% interval say, I would expect my interval to contain the truth 95% of the time, correct?

Thank you!

Answer 1

1. Yes, it is perfectly sensible.
2. For a quick interpretation, I like the one provided by Davison: Assuming $T$ is an estimator of a parameter $\psi$ based on a random sample $Y_1, . . . , Y_n$, $V_T^{0.5}$ is the standard error of $T$, $n \rightarrow \infty$ and $\zeta_\alpha$ is the $\alpha$-th quantile of a standard normal distribution function, the interval with endpoints: $T − \zeta_{1−\alpha} V_T^{0.5}, T + \zeta_{\alpha} V_T^{0.5}$ contains $\psi_0$, the true but unknown value of $\psi$, with probability approximately $(1 − 2\alpha)$. (See A.C. Davison "Statistical models", Chapt. 3 for more.)

These said, there are some insightful threads in CV as why this is a somewhat over-simplified view of what a confidence interval is: "Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?", "Interpretation of confidence interval" and "Is it true that the percentile bootstrap should never be used?". To quote Hastie et al. from the book "Elements of Statistical Learning" (Sect. 8.4) directly: "we might think of the bootstrap distribution as a "poor man's" Bayes posterior. By perturbing the data, the bootstrap approximates the Bayesian effect of perturbing the parameters, and is typically much simpler to carry out."

When bootstrapping I think is much more important not to forget accounting for dependence structures (e.g. as described for example in Owen and Eckles' Bootstrapping data arrays of arbitrary order). These might be due to clustering of the data, heteroskedasticity (e.g. see the notion of wild bootstrap) and other deviations from IID data generating procedures. Ignoring such issues will render any discussions about the subsequent interpretation of the generated CIs, moot.