3
$\begingroup$

I have recently been bootstrapping the confidence intervals of a neural network model estimated to data.

I execute the following psudo-code, which seems similar to previous bootstraps I have done:

  1. Take all observations of data $x_i$, estimate $\hat{y} = \hat{\theta}(x_i)$
  2. Resample $N$ observations from the set of $x_i$ with replacement. Estimate $\tilde{y} = \tilde{\theta}(x_i)$. Repeat until we have $M$ different versions of $\tilde{y}_m$
  3. Calculate the standard deviation of everything in $\tilde{y}_M$ as $\sigma_{\tilde{y}}$ and then the confidence interval of $\hat{y}$ is $\hat{y} \pm z * \sigma_{\tilde{y}}$ where z is chosen to be an appropriate value from a standard normal distribution, usually 1.96 for 95% CI.

See here for more reading on bootstrap intervals.

At the same time, I see other approaches that use the quantiles of $\tilde{y}_M$ in order to construct intervals of some sort, such as here and here. They do this instead of the $\hat{y} \pm z * \sigma_{\tilde{y}}$ I have come to expect. The intuition for such an argument is very strong and seems valid - but the approach is unfamiliar. What's going on with the use of quantiles? Is something else being calculated (prediction intervals, etc.) that are similar in flavor but are not the same?

$\endgroup$
1
  • 2
    $\begingroup$ Davison and Hinkley's book Bootstrap Methods and their Application presents a number of approaches to bootstrapping (not exhaustive but both parametric and nonparametric cases are discussed) and it does discuss bootstrap prediction intervals and nonparametric regression models; this might be helpful as a reference, for all that the various pieces are not all in the same places, since the relevant concepts should carry across. $\endgroup$
    – Glen_b
    May 12, 2022 at 1:50

1 Answer 1

7
$\begingroup$

A possible alternative is to identify and form an efficient model generating a set of iid residuals. Generate a forecast using the model and then use the distribution of residuals as the basis of the montecarlo creating limits that are not presumptively based upon a distributional assumption. Use this empirical distribution to offset the expected value. This method allows the identification and incorporation of possible unusual pulses and the option to incorporate (allow for) predicted pulses into the forecast distribution. This is the suggested approach of Prpf. Sam Savage as an extension to his book "The Flaw of Averages".

$\endgroup$
5
  • $\begingroup$ I'm a little confused - ARMAX modeling? I am asking about when it is appropriate to use $\pm z*SD$ vs quantiles to estimate confidence intervals. $\endgroup$ May 11, 2022 at 15:17
  • 1
    $\begingroup$ OP: @IrishStat is a time series guru passionate about forecasting in time and is guessing that that is what you are doing. $\endgroup$
    – Nick Cox
    May 11, 2022 at 15:19
  • $\begingroup$ i pulled the refernce to ARMAX $\endgroup$
    – IrishStat
    May 11, 2022 at 15:19
  • $\begingroup$ Past and future pulses remain.... $\endgroup$
    – Nick Cox
    May 11, 2022 at 15:20
  • $\begingroup$ Thanls Nick although in my view cross-sectional data is just a specific case of time series data. . Past and future one-time anomalies could remain whether or not the data is time series $\endgroup$
    – IrishStat
    May 11, 2022 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.