# Can we make probabilistic statements with prediction intervals?

I've read through the many excellent discussions on the site regarding interpretation of confidence intervals and prediction intervals, but one concept is still a bit puzzling:

Consider the OLS framework and we've obtained the fitted model $\hat y = X\hat\beta$. We're given a $x^*$ and asked to predict its response. We compute $x^{*T}\hat\beta$ and, as a bonus, we also provide a 95% prediction interval around our prediction, a la Obtaining a formula for prediction limits in a linear model. Let's call this prediction interval PI.

Now, which of the following (or neither) is the correct interpretation of PI?

1. For $x^*$ in particular, $y(x^*)$ lies within PI with 95% probability.
2. If we're given a large number of $x$s, this procedure to compute PIs will cover the true responses 95% of the time.

From @gung's wording in Linear regression prediction interval, it seems like the former is true (although I could very well be misinterpreting.) Interpretation 1 seems counterintuitive to me (in the sense that we're drawing Bayesian conclusions from frequentist analysis), but if it's correct, is it because we're predicting the realization of a random variable vs. estimating a parameter?

(Edit) Bonus question: Suppose we knew what the true $\beta$ is, i.e. the process generating the data, then would we be able talk about probabilities regarding any particular prediction, since we're just looking at $\epsilon$?

My latest attempt at this: we can "conceptually decompose" (using the word very loosely) a prediction interval into two parts: (A) a confidence interval around the predicted mean response, and (B) a collection of intervals which are just quantile ranges of the error term. (B) we can make probabilistic statements on, conditional on knowing the true predicted mean, but as a whole, we can only treat prediction intervals as frequentist CIs around predicted values. Is this somewhat correct?

• The answer I wrote at stats.stackexchange.com/a/26704 implies that something like (2) is the case (according to laws of large numbers) but definitely not (1).
– whuber
May 19, 2014 at 15:02

First, on the use of the word probability, frequentists don't have a problem with using the word probability when predicting something where the random piece has not taken place yet. We don't like the word probability for a confidence interval because the true parameter is not changing (we are assuming it is a fixed, though unknown, value) and the interval is fixed because it is based on data that we have already collected. For example if our data comes from a random sample of adult male humans and x is their height and y is their weight and we fit the general regression model then we don't use probability when talking about the confidence intervals. But if I want to talk about what is the probability of a 65 inch tall male chosen at random from all 65 inch tall males having a weight within a certain interval, then it is fine to use probability in that context (because the random selection has not yet been made, so probability makes sense).

So I would say that the answer to the bonus question is "Yes". If we knew enough information, then we could compute the probability of seeing a y value within an interval (or find an interval with the desired probability).

For your statement labeled "1." I would say that it is OK if you use a word like "approximate" when talking about the interval or probability. Like you mention in the bonus question, we can decompose the uncertainty into a piece about the center of the prediction and a piece about the randomness around the true mean. When we combine these to cover all our uncertainty (and assuming we have the model/normality correct) we have an interval that will tend to be too wide (though can be too narrow as well), so the probability of a new randomly chosen point falling into the prediction interval is not going to be exactly 95%. You can see this by simulation. Start with a known regression model with all the parameters known. Choose a sample (across many x values) from this relationship, fit a regression, and compute the prediction interval(s). Now generate a large number of new data points from the true model again and compare them to the prediction intervals. I did this a few times using the following R code:

x <- 1:25
y <- 5 + 3*x + rnorm(25, 0, 5)
plot(x,y)

fit <- lm(y~x)
tmp <- predict(fit, data.frame(x=1:25), interval='prediction')

sapply( 1:25, function(x){
y <- rnorm(10000, 5+3*x, 5)
mean( tmp[x,2] <= y & y <= tmp[x,3] )
})


I ran the above code a few times (around 10, but I did not keep careful count) and most of the time the proportion of new values falling in the intervals ranged in the 96% to 98% range. I did have one case where the estimated standard deviation was very low that the proportions were in the 93% to 94% range, but all the rest were above 95%. So I would be happy with your statement 1 with the change to "approximately 95%" (assuming all the assumptions are true, or close enough to be covered in the approximately).

Similarly, statement 2 needs an "approximately" or similar, because to cover our uncertainty we are capturing on average more than the 95%.

The second is better. The first depends on what other information is known.

Using a random example, it is true that "95% of intervals (at 95% confidence) would include the true mean of [insert variable]".

On the other hand, if a result is obviously counter-intuitive, we cannot assert (1).

E.g., "my significance test at 95% confidence shows that height and weight are negatively correlated". Well that's obviously false, and we cannot say that there is a "95% probability that it is true". There is in fact, taking into consideration prior knowledge, a very small probability that it is true. It is, however, valid to say that "95% of such tests would have yielded a correct result."

• This answer seems to discuss confidence intervals rather than prediction intervals.
– whuber
May 19, 2014 at 15:03
• @whuber The same principle applies. We are essentially dealing with confidence intervals for a certain variable (the "predicted" variable).
– user45240
May 19, 2014 at 15:06
• There is an important distinction between a fixed value (like a parameter) and the value of a random variable. Moreover, the heart of the present question gets to this distinction: what can be said about the probability of that ("future") random outcome? It therefore appears inadequate--and possibly misleading--to treat this question as being one merely about the meaning of confidence.
– whuber
May 19, 2014 at 15:11
• @whuber The statement (2) in the post still does not imply statement (1). As in my example, a prediction which went against obvious intuition/background knowledge would not imply that future outcomes have 95% chance of falling in the PI. It is true that the process, 95% of the time, would give PI's containing the future outcome. But it is sometimes possible to detect when this has or has not happened.
– user45240
May 19, 2014 at 15:16
• You are right, but if I'm reading your comment correctly I suspect it misses the point. The issue is not the fact that (by design) a PI has only a 95% chance of covering the future value or that additional data (or intuition) could give more information. The matter before us concerns whether a PI can be interpreted in terms of a conditional probability for the future value (based on the regression values). That indeed is the interpretation of a Bayes PI, as the O.P. notes, but it is invalid for a frequentist PI.
– whuber
May 19, 2014 at 15:23