# Is there a general and model-independent way of calculating prediction intervals in machine learning for regression task?

I’m training some supervised machine learning algorithm to perform the prediction of a continuous variable.

I’m currently applying a nested cross-validation protocol (inner: LOOCV; outer: LOOCV; sample). It is a pilot and I have only 70 cases at the moment, while a test may come later if the pilot goes well.

I’m looking for a general strategy (aka applicable to any supervized technique, potentially also to an ensamble of them) to construct prediction intervals both for each current outer loop predictions as well as for future predictions of new subjects not included in the current sample.

In the general regression case, problem formulation is assumed to be something like this $$Y = \mu ( X ) + e,$$ where $\mu$ is some arbitrary (but preferably smooth) transformation of the data and $e$ is some independent noise term. Doing the model based resampling, you start with $$y_j^* = \hat{\mu}(x_j) + e^*_j$$
where $\hat{\mu}(x_j )$ is your fitted model values. Then you can calculate the residuals from the fitted model $y_j - \hat{\mu}(x_j)$. By appropriately modifying these residuals you can then sample errors $e^*_j$ to generate your prediction interval. Note this procedure is naive, and the estimates of the residuals will likely be biased from the assumed functional form of $\mu$. This is discussed in more detail and more thoughtfully in Section 7.6 of Davidson and Hinckley (1997), Bootstrap Methods and Their Application.