3
$\begingroup$

I want to come up with a way to get how confident I am in my predictions. I am not using a Bayesian model so I was thinking a bootstrap confidence interval would be good:

I would re-sample my original training set, train on it and predict. And repeat that n number of times. I would then gather the n predictions and I could get some bootstrap intervals on my prediction.

The way that I interpret that would be: the larger the interval, the more sensitive my prediction is to the variation in the training data, which would in turn mean that I am fairly uncertain about my prediction. The smaller the interval would mean the opposite.

The questions are:

1) Is this a sensible thing to do to start with? 2) Is there any other interpretation that could be made? In this case, it is not true that for a 95% interval say, I would expect my interval to contain the truth 95% of the time, correct?

Thank you!

$\endgroup$
6
+25
$\begingroup$
  1. Yes, it is perfectly sensible.
  2. For a quick interpretation, I like the one provided by Davison: Assuming $T$ is an estimator of a parameter $\psi$ based on a random sample $Y_1, . . . , Y_n$, $V_T^{0.5}$ is the standard error of $T$, $n \rightarrow \infty$ and $\zeta_\alpha$ is the $\alpha$-th quantile of a standard normal distribution function, the interval with endpoints: $T − \zeta_{1−\alpha} V_T^{0.5}, T + \zeta_{\alpha} V_T^{0.5}$ contains $\psi_0$, the true but unknown value of $\psi$, with probability approximately $(1 − 2\alpha)$. (See A.C. Davison "Statistical models", Chapt. 3 for more.)

These said, there are some insightful threads in CV as why this is a somewhat over-simplified view of what a confidence interval is: "Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?", "Interpretation of confidence interval" and "Is it true that the percentile bootstrap should never be used?". To quote Hastie et al. from the book "Elements of Statistical Learning" (Sect. 8.4) directly: "we might think of the bootstrap distribution as a "poor man's" Bayes posterior. By perturbing the data, the bootstrap approximates the Bayesian effect of perturbing the parameters, and is typically much simpler to carry out."

When bootstrapping I think is much more important not to forget accounting for dependence structures (e.g. as described for example in Owen and Eckles' Bootstrapping data arrays of arbitrary order). These might be due to clustering of the data, heteroskedasticity (e.g. see the notion of wild bootstrap) and other deviations from IID data generating procedures. Ignoring such issues will render any discussions about the subsequent interpretation of the generated CIs, moot.

$\endgroup$
  • 3
    $\begingroup$ I think the focus on confidence intervals - rather than prediction intervals - might be misplaced. $\endgroup$ – Richard Hardy Sep 3 '18 at 17:50
  • $\begingroup$ @RichardHardy Thank you for your comment. I see why one may comment that but I think that when it comes to bootstrap-generated intervals the difference between the two is immaterial. Obviously PIs account for variance due to error terms and yes, we might want to use variance-adjusted residuals for a parametric bootstrap, but a simple non-parametric bootstrap (as described by the OP) is accounting for that during bootstrap estimation. The final statistic incorporates the error. (Apologies for the late reply, I was away for a while) $\endgroup$ – usεr11852 says Reinstate Monic Sep 10 '18 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.