Prediction Interval for Mean of Predictions

Question

This question is about creating a prediction interval for the mean of predictions from a regressor.

Let's say I have arbitrary regression function (not necessarily parametric, could be random forest, etc.) $f: X\mapsto\hat{y}$ which yields predictions $\hat{y} = f(X)$. I also know $y_{true}$ for a training set, which allows me to calculate an estimate for $R^2$ score, $MSE$, etc. However, I'm not actually interested in the individual $\hat{y}$ values, but rather their mean, $mean(\hat{y})$.

Is there any way to use my comparison between $\hat{y}$ and $y_{true}$ above for the training set to then estimate a prediction interval for $mean(\hat{y})$ for a testing set where $y_{true}$ is unknown? My intuition is that the prediction interval for the mean of predictions should be narrower than the interval for an individual prediction, but I'm not sure how move from this intuition into an actual prediction interval, especially on a non-parametric model. Any help would be appreciated.

Answer 1

Accidently I've the same type of problem: Although my objective is to create a prediction interval for the sum of predictions $\hat{Y}$ instead of the mean of predicted values.

I also found out there's less literature about this topic (just on prediction intervals for point predictions of nonparametric methods such as this one: Bootstrap prediction interval, although you can also use quantile regression techniques in this case).

What I've tried to do is the following.

For $i=1$ to $N$ bootstrapped versions of the testset:

Make individual predictions $\hat{Y}$ on the (labeled!) testset and aggegrate (e.g. averaging or summing) these predictions to $\hat{Y_{tot}}$. Now calculate $Y_{tot}$ (which is computable since the individual $Y$ values in the testset are also known). Then write the relative error $e_i=\frac{\hat{Y_{tot}}-Y_{tot}}{Y_{tot}}$.

Now extract the $\frac{1-\alpha}{2}$ and $\frac{1+\alpha}{2}$ quantiles of the vector of ordered $e_i$'s, this gives you insight in the $\alpha$% prediction interval for some new $Y_{tot}$; namely $P(\frac{\hat{Y_{tot}}}{1+e_{\frac{1+\alpha}{2}}} \leq Y_{tot} \leq \frac{\hat{Y_{tot}}}{1+e_{\frac{1-\alpha}{2}}})=\alpha$

Although, theoretically, this methods seems valid I still have problems regarding stability and convergence which I don't manage to solve (for example varying error bounds when my testset varies in size)...