Prediction uncertainty intervals for predictions of machine learning algorithms

Question

Assume I have a regression problem. I fit models on a train data set and tune their hyperparameters using CV. I then run the models on the test set.

What is the best way to calculate prediction intervals for the predicted values? I.e. for a new data point (inputs X), my GBM model outputs the predicted value Y. However, I'd like to know the interval which will contain 95% of the time the true value of the predicted variable given these input features, not just the point estimate Y.

1. Assume that my predictors and the target are known precisely (have no errors). In that case could the distribution of errors from the test sample could be used to calculate say the 95% interval on any point estimate that the model outputs. This also assumes that the errors are homoscedastic. Is this correct? Any further assumptions are needed?

Or to put it even more bluntly: is the test sample error distribution my estimate of the posterior for any given point estimate output of the model? I don't think that can be quite true...

2. If my predictors have errors, I'd probably simulate observations drawn around the input values and add the interval in square to the one from step 1.

3. If also my target has an error, I'd likely do the above but already during the learning phase, sampling also for the targets. Then add the square error further on top. This sounds too complicated to be really practical...

Would these procedures be right? For linear models there are shortcuts, but for more complex stuff like ANN/SVM/GBM/etc.? Also what about the distinction whether the model estimates E(target|predictors) vs. directly target|predictors.

I think we actually want posteriors for any model output, but I haven't seen this done anywhere (I guess online learning likely does it, but outside that area?).

Any pointers? Thank you very much!

Answer 1

Unlike the classical statistical literature, typical machine learning methods do not assume the data generating process (aka the sampling distribution or the generative model). I thus suggest quantile estimation: You could try learning the $\alpha/2$ and $1-\alpha/2$ quantile of the conditional distribution. The intervals between these, can be seen as a $1-\alpha$ prediction interval.