Is there a general and model-independent way of calculating prediction intervals in machine learning for regression task?

Question

I’m training some supervised machine learning algorithm to perform the prediction of a continuous variable.

I’m currently applying a nested cross-validation protocol (inner: LOOCV; outer: LOOCV; sample). It is a pilot and I have only 70 cases at the moment, while a test may come later if the pilot goes well.



I’m looking for a general strategy (aka applicable to any supervized technique, potentially also to an ensamble of them) to construct prediction intervals both for each current outer loop predictions as well as for future predictions of new subjects not included in the current sample.

Answer 1

Yes, there are general strategies available to you based on other resampling techniques. In particular, bootstrapping either the fitted model residuals or cases in the dataset will let you construct prediction intervals. I think it's recommend to bootstrap the residuals rather than cases because there can often be a problem with bias. The procedure for bootstrapping the residuals is outlined in detail for linear models in the answer to this question although it is applicable more widely.

In the general regression case, problem formulation is assumed to be something like this $$ Y = \mu ( X ) + e, $$ where $\mu$ is some arbitrary (but preferably smooth) transformation of the data and $e$ is some independent noise term. Doing the model based resampling, you start with $$ y_j^* = \hat{\mu}(x_j) + e^*_j $$
where $\hat{\mu}(x_j )$ is your fitted model values. Then you can calculate the residuals from the fitted model $y_j - \hat{\mu}(x_j)$. By appropriately modifying these residuals you can then sample errors $e^*_j$ to generate your prediction interval. Note this procedure is naive, and the estimates of the residuals will likely be biased from the assumed functional form of $\mu$. This is discussed in more detail and more thoughtfully in Section 7.6 of Davidson and Hinckley (1997), Bootstrap Methods and Their Application.