3
$\begingroup$

I’m training some supervised machine learning algorithm to perform the prediction of a continuous variable.

I’m currently applying a nested cross-validation protocol (inner: LOOCV; outer: LOOCV; sample). It is a pilot and I have only 70 cases at the moment, while a test may come later if the pilot goes well.



I’m looking for a general strategy (aka applicable to any supervized technique, potentially also to an ensamble of them) to construct prediction intervals both for each current outer loop predictions as well as for future predictions of new subjects not included in the current sample.

$\endgroup$
2
$\begingroup$

Yes, there are general strategies available to you based on other resampling techniques. In particular, bootstrapping either the fitted model residuals or cases in the dataset will let you construct prediction intervals. I think it's recommend to bootstrap the residuals rather than cases because there can often be a problem with bias. The procedure for bootstrapping the residuals is outlined in detail for linear models in the answer to this question although it is applicable more widely.

In the general regression case, problem formulation is assumed to be something like this $$ Y = \mu ( X ) + e, $$ where $\mu$ is some arbitrary (but preferably smooth) transformation of the data and $e$ is some independent noise term. Doing the model based resampling, you start with $$ y_j^* = \hat{\mu}(x_j) + e^*_j $$
where $\hat{\mu}(x_j )$ is your fitted model values. Then you can calculate the residuals from the fitted model $y_j - \hat{\mu}(x_j)$. By appropriately modifying these residuals you can then sample errors $e^*_j$ to generate your prediction interval. Note this procedure is naive, and the estimates of the residuals will likely be biased from the assumed functional form of $\mu$. This is discussed in more detail and more thoughtfully in Section 7.6 of Davidson and Hinckley (1997), Bootstrap Methods and Their Application.

$\endgroup$
  • $\begingroup$ Thank you @MachineEpsilon ! The procedure for (unregularized) linear models was very clearly explained in the answer you pointed me to, but I'm not fully sure that it may/I don't fully understand how it can be translated to other techniques $\endgroup$ – Massimiliano Grassi Feb 24 '18 at 11:31
  • $\begingroup$ That's fair. I will try to elaborate on what I suggested. $\endgroup$ – MachineEpsilon Feb 25 '18 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.