2
$\begingroup$

This question is about creating a prediction interval for the mean of predictions from a regressor.

Let's say I have arbitrary regression function (not necessarily parametric, could be random forest, etc.) $f: X\mapsto\hat{y}$ which yields predictions $\hat{y} = f(X)$. I also know $y_{true}$ for a training set, which allows me to calculate an estimate for $R^2$ score, $MSE$, etc. However, I'm not actually interested in the individual $\hat{y}$ values, but rather their mean, $mean(\hat{y})$.

Is there any way to use my comparison between $\hat{y}$ and $y_{true}$ above for the training set to then estimate a prediction interval for $mean(\hat{y})$ for a testing set where $y_{true}$ is unknown? My intuition is that the prediction interval for the mean of predictions should be narrower than the interval for an individual prediction, but I'm not sure how move from this intuition into an actual prediction interval, especially on a non-parametric model. Any help would be appreciated.

$\endgroup$
  • 1
    $\begingroup$ A prediction interval is not the same as a confidence interval. It takes into account that extra uncertainty due to an unknown new observation. $\endgroup$ – Michael R. Chernick May 7 '18 at 14:56
  • $\begingroup$ To be clear: you want a prediction interval, rather than a confidence interval for the mean of $\hat{y}$? And are you specifically looking to avoid bootstrap methods? I haven't thought deeply about this, but what about simply using a nonparametric bootstrap, and looking at the quantiles of $E[\hat{y}]$? $\endgroup$ – generic_user May 7 '18 at 15:05
  • $\begingroup$ If using a random forest, this would be simplified by using the out-of-bag samples, akin to the jackknife after bootstrap. But I hasten to add that you need to think this through more carefully than I have. $\endgroup$ – generic_user May 7 '18 at 15:06
  • $\begingroup$ @MichaelChernick thanks for the point - amending language accordingly $\endgroup$ – Owen May 7 '18 at 16:17
  • 3
    $\begingroup$ See Hahn & Meeker Statistical Intervals for an explanation of prediction intervals of a mean of future samples that readily extends to regression. $\endgroup$ – whuber May 26 '18 at 13:22
0
$\begingroup$

Accidently I've the same type of problem: Although my objective is to create a prediction interval for the sum of predictions $\hat{Y}$ instead of the mean of predicted values.

I also found out there's less literature about this topic (just on prediction intervals for point predictions of nonparametric methods such as this one: Bootstrap prediction interval, although you can also use quantile regression techniques in this case).

What I've tried to do is the following.

For $i=1$ to $N$ bootstrapped versions of the testset:

Make individual predictions $\hat{Y}$ on the (labeled!) testset and aggegrate (e.g. averaging or summing) these predictions to $\hat{Y_{tot}}$. Now calculate $Y_{tot}$ (which is computable since the individual $Y$ values in the testset are also known). Then write the relative error $e_i=\frac{\hat{Y_{tot}}-Y_{tot}}{Y_{tot}}$.

Now extract the $\frac{1-\alpha}{2}$ and $\frac{1+\alpha}{2}$ quantiles of the vector of ordered $e_i$'s, this gives you insight in the $\alpha$% prediction interval for some new $Y_{tot}$; namely $P(\frac{\hat{Y_{tot}}}{1+e_{\frac{1+\alpha}{2}}} \leq Y_{tot} \leq \frac{\hat{Y_{tot}}}{1+e_{\frac{1-\alpha}{2}}})=\alpha$

Although, theoretically, this methods seems valid I still have problems regarding stability and convergence which I don't manage to solve (for example varying error bounds when my testset varies in size)...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.