# finding prediction interval for any type of regression

I am working on a project where I need to estimate some real-valued values given some features in an online fashion. I evaluate various online learning methods. My estimations has to be bounded by a prediction interval. I know that not every regression model is suitable for coming up with prediction intervals. For instance, non-probabilistic models such as regression-trees are not good for this. I wonder if there is any technique which can help me find prediction intervals only from the predicted data and the error(response-prediction) regardless of the learning method used? If not, what regression techniques should I look at if I strictly need prediction intervals around my predictions?

• Are you interested in confidence intervals for parameters or in prediction intervals for individual predictions? – Michael M Apr 1 '15 at 15:04
• confidence intervals for individual predictions – bfaskiplar Apr 1 '15 at 15:12
• Hmm, it is no option to combine the two options in my comment... – Michael M Apr 1 '15 at 19:11
• I meant the latter..prediction intervals for individual predictions – bfaskiplar Apr 1 '15 at 19:28

If the data set is large, you can always get naive $(1-\alpha)\cdot 100\%$ prediction intervals by adding to the point predictions $\hat Y$ the empirical $\alpha/2$ and $1-\alpha/2$ quantiles $\hat Q_{\alpha/2}$ and $\hat Q_{1-\alpha/2}$ of the residuals.

They are naive

1. because you ignore the variability in the point prediction (viewed as an estimator of a population average) and
2. because the distribution of the residuals might differ depending on the input values.

Both issues can usually be mended in the concrete case (e.g. calculate the quantiles separately per leaf of the tree to allow non-homogenous residual distributions) but there is no general solution.

Here a quick illustration in R to compare such naive 50%-prediction interval with the "classic" one:

fit <- lm(Sepal.Width ~ ., iris)

# Implemented prediction interval for first obs
predict(fit, iris[1, ], interval = "p", level = 0.5)

#       fit      lwr      upr
# 3.446357 3.263315 3.629399

# Naive prediction interval described as above
predict(fit, iris[1, ]) + quantile(resid(fit), c(0.25, 0.75))

#      25%      75%
# 3.298499 3.631797


In this example, the "classic" interval almost agrees with the naive one based on empirical quantiles.