0
$\begingroup$

I am reading "Confidence Intervals and Prediction Intervals for Feed-Forward Neural Networks" by Richard Dybowski. In this paper, an ensemble of neural networks are trained on bootstrapped data samples so that a confidence interval can be estimated. I will try to briefly summarize the method below:

Assume that our training data is modeled by the stochastic function

$$ y = \mu_y(\mathbf{x}) + \epsilon, \quad y\in\mathbb{R}, \mathbf{x}\in\mathbb{R}^m$$

where $\mu_y$ is the unknown function to be approximated.

We have access to a finite sample of $N$ training pairs $\mathbf{S} = \{(\mathbf{x}^{(1)}, y^{(1)}), \dots, (\mathbf{x}^{(N)}, y^{(N)})\}$ only.

We generate $B$ bootstrapped samples $\{ (\mathbf{x}^{(*b, 1)}, y^{(*b, 1)}), \dots, (\mathbf{x}^{(*b, N)}, y^{(*b, N)}) \}_{b=1}^B$, and train $B$ neural networks $\{\hat{\mu}_y(\mathbf{x}; \hat{\mathbf{w}}^{(*b)})\}_{b=1}^B$.

Then the bootstrap estimate of $\mu_y(\mathbf{x})$ is given by the mean provided by the ensemble of neural networks

$$ \hat{\mu}_{y, \text{boot}}(\mathbf{x}) = \frac{1}{B} \sum_{b=1}^B \hat{\mu_y}(\mathbf{x}; \hat{\mathbf{w}}^{(*b)} )$$

and the bootstrap estimate of the standard error of $\hat{\mu}_y(\mathbf{x}; \hat{\mathbf{w}})$ is given by

$$\hat{\mathrm{SE}}_\text{boot} (\hat{\mu}_y (\mathbf{x}; \cdot)) = \sqrt{ \frac{1}{B-1} \sum_{b=1}^B [ \hat{\mu}_y ( \mathbf{x}; \hat{\mathbf{w}}^{(*b)} ) - \hat{\mu}_{y, \text{boot}}(\mathbf{x})]^2 }$$

Assuming a normal distribution for $\hat{\mu}_y ( \mathbf{x}; \hat{\mathbf{w}} )$ over the space of all possible $\hat{\mathbf{w}}$, we have

$$\hat{\mu}_{y, \text{boot}}(\mathbf{x}) \pm t_{.025}\hat{\mathrm{SE}}_\text{boot}(\hat{\mu}_y(\mathbf{x}; \cdot))$$

as the 95% confidence interval for $\mu_y(\mathbf{x})$.

1. How would I compute the 95% confidence interval without assuming a normal distribution for $\hat{\mu}_y ( \mathbf{x}; \hat{\mathbf{w}} )$?

2. How would I generalise this method to a vector output $\mathbf{y} \in \mathbb{R}^m$?

$\endgroup$
2
  • $\begingroup$ You state "Any regression function μ^y(x) trained on S would suffer from sampling error." but do not explain. This isn't necessarily the case. Why do you believe your sample is biased? Bootstrapping will only exacerbate whatever problem is with the original sample if you don't have a probability mechanism to explain any possible bias in the sample. And if you do have such a probability mechanism, then it's unclear why a bootstrap is needed. $\endgroup$
    – AdamO
    Commented Apr 28, 2023 at 17:33
  • $\begingroup$ I'm not sure I understand your comment, but I've edited my question to add some further context. $\endgroup$
    – Jacob
    Commented Apr 28, 2023 at 20:33

1 Answer 1

2
$\begingroup$
  1. You get it by using the percentile method. Got 95% ci will be specified by 2.5 and 97.5 percentiles of your bootstrap distribution. In general this will need much more bootstrap replications than the standard error method. In Efron's bootstrap book it is said that for SE estimation you need 20-200 replications whole for confidence intervals you need thousands.

  2. You just calculate ci for each variable in your vector separately. You can also get something like a confidence region by taking covariance of these predictions into account, but i don't know how it what it would be useful for, but it should not be difficult to do

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.