1
$\begingroup$

Assume I have a regression problem. I fit models on a train data set and tune their hyperparameters using CV. I then run the models on the test set.

What is the best way to calculate prediction intervals for the predicted values? I.e. for a new data point (inputs X), my GBM model outputs the predicted value Y. However, I'd like to know the interval which will contain 95% of the time the true value of the predicted variable given these input features, not just the point estimate Y.

  1. Assume that my predictors and the target are known precisely (have no errors). In that case could the distribution of errors from the test sample could be used to calculate say the 95% interval on any point estimate that the model outputs. This also assumes that the errors are homoscedastic. Is this correct? Any further assumptions are needed?

Or to put it even more bluntly: is the test sample error distribution my estimate of the posterior for any given point estimate output of the model? I don't think that can be quite true...

  1. If my predictors have errors, I'd probably simulate observations drawn around the input values and add the interval in square to the one from step 1.

  2. If also my target has an error, I'd likely do the above but already during the learning phase, sampling also for the targets. Then add the square error further on top. This sounds too complicated to be really practical...

Would these procedures be right? For linear models there are shortcuts, but for more complex stuff like ANN/SVM/GBM/etc.? Also what about the distinction whether the model estimates E(target|predictors) vs. directly target|predictors.

I think we actually want posteriors for any model output, but I haven't seen this done anywhere (I guess online learning likely does it, but outside that area?).

Any pointers? Thank you very much!

$\endgroup$
  • 1
    $\begingroup$ Confidence intervals are constructed for model parameters. Do you mean Prediction Intervals? $\endgroup$ – JohnRos Apr 25 '15 at 18:43
  • $\begingroup$ I didn't know that these are called prediction intervals. Yes, I'm interested in the uncertainty of the predicted values not the parameters of the model. I edited the question, hopefully it is not confusing now. $\endgroup$ – user673592 Apr 25 '15 at 20:59
  • $\begingroup$ Also googling now "prediction intervals" seems to give some useful leads. Sometimes one really just misses the correct words... I'd leave the question open, maybe somebody can give good literature pointers or direct answers to the question. $\endgroup$ – user673592 Apr 25 '15 at 21:03
2
$\begingroup$

Unlike the classical statistical literature, typical machine learning methods do not assume the data generating process (aka the sampling distribution or the generative model). I thus suggest quantile estimation: You could try learning the $\alpha/2$ and $1-\alpha/2$ quantile of the conditional distribution. The intervals between these, can be seen as a $1-\alpha$ prediction interval.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.