21
$\begingroup$

Let's take the following example:

set.seed(342)
x1 <- runif(100)
x2 <- runif(100)
y <- x1+x2 + 2*x1*x2 + rnorm(100)
fit <- lm(y~x1*x2)

This creates a model of y based on x1 and x2, using a OLS regression. If we wish to predict y for a given x_vec we could simply use the formula we get from the summary(fit).

However, what if we want to predict the lower and upper predictions of y? (for a given confidence level).

How then would we build the formula?

Improve this question
$\endgroup$
7
  • $\begingroup$ The Confidence Interval on New Observations section of this page may help. $\endgroup$
    – GaBorgulya
    Apr 3, 2011 at 18:32
  • $\begingroup$ @Tal Sorry, but it's not really clear to me what you actually mean by "predict the lower and upper predictions of y". Does it have something to do with prediction or tolerance bands? $\endgroup$
    – chl
    Apr 3, 2011 at 19:43
  • $\begingroup$ @Tal - a couple of queries. When you say " .. y based on x1 and x2, using a OLS regression." , you mean your create a linear model and estimate parameters using OLS. Am I right? and @chl's question -- do you want to predict the lower and upper bounds for the prediction interval? $\endgroup$
    – suncoolsu
    Apr 3, 2011 at 19:55
  • $\begingroup$ @chl, sorry for not being more clear. I am looking for two formulas that will give an interval for that will "catch" the "real" value of y 95% of the time. I feel how I'm using definitions for the CI for the mean, when there is probably some other term I should be using, sorry about that... $\endgroup$
    – Tal Galili
    Apr 3, 2011 at 20:02
  • $\begingroup$ @suncoolsu - yes and yes. $\endgroup$
    – Tal Galili
    Apr 3, 2011 at 20:03

3 Answers 3

Reset to default
29
$\begingroup$

You will need matrix arithmetic. I'm not sure how Excel will go with that. Anyway, here are the details.

Suppose your regression is written as $\mathbf{y} = \mathbf{X}\mathbf{\beta} + \mathbf{e}$.

Let $\mathbf{X}^*$ be a row vector containing the values of the predictors for the forecasts (in the same format as $\mathbf{X}$). Then the forecast is given by $$ \hat{y} = \mathbf{X}^*\hat{\mathbf{\beta}} = \mathbf{X}^*(\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y} $$ with an associated variance $$ \sigma^2 \left[1 + \mathbf{X}^* (\mathbf{X}'\mathbf{X})^{-1} (\mathbf{X}^*)'\right]. $$ Then a 95% prediction interval can be calculated (assuming normally distributed errors) as $$ \hat{y} \pm 1.96 \hat{\sigma} \sqrt{1 + \mathbf{X}^* (\mathbf{X}'\mathbf{X})^{-1} (\mathbf{X}^*)'}. $$ This takes account of the uncertainty due to the error term $e$ and the uncertainty in the coefficient estimates. However, it ignores any errors in $\mathbf{X}^*$. So if the future values of the predictors are uncertain, then the prediction interval calculated using this expression will be too narrow.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ +1, excellent answer. I should note though, that regression model always estimates conditional expectation, so it is as good as its regressors are. So the last comment although is very good, it is not strictly necessary, since if you build regression model you must trust the regressors. $\endgroup$
    – mpiktas
    Apr 4, 2011 at 3:51
  • 1
    $\begingroup$ why the 1 comes up in the formula? We have $\hat{y}=X^*\beta+X^*(X'X)^{-1}X'e$. Then $var \hat{y}=var X^*(X'X)^{-1}X'e=\sigma^2X^*(X'X)^{-1}(X^*)'$? $\endgroup$
    – mpiktas
    Apr 7, 2011 at 14:25
  • 1
    $\begingroup$ The 1 is for prediction intervals. Leave it off for confidence intervals. Var($\hat{y}$) relates to confidence intervals. $\endgroup$
    – Rob Hyndman
    Apr 12, 2011 at 14:44
  • $\begingroup$ @RobHyndman thank you for your excelent answer (one year ago ;)) however, am I missing something or is the term in the square root $N \times N$? $\endgroup$
    – Seb
    Sep 28, 2012 at 6:06
  • 1
    $\begingroup$ I did not provide the variance of y-hat, but the forecast variance. $\endgroup$
    – Rob Hyndman
    Nov 25, 2021 at 1:19
9
$\begingroup$

Are you by chance after the different types of prediction intervals? The predict.lm manual page has

 ## S3 method for class 'lm'
 predict(object, newdata, se.fit = FALSE, scale = NULL, df = Inf, 
         interval = c("none", "confidence", "prediction"),
         level = 0.95, type = c("response", "terms"),
         terms = NULL, na.action = na.pass,
         pred.var = res.var/weights, weights = 1, ...)

and

Setting ‘intervals’ specifies computation of confidence or prediction (tolerance) intervals at the specified ‘level’, sometimes referred to as narrow vs. wide intervals.

Is that what you had in mind?

Improve this answer
$\endgroup$
5
  • $\begingroup$ Hi Dirk, that is indeed what I wish to find, but I want the upper and lower bonds to be in the form of a formula (so to later implement in some low form of statistical software, for example, excel...) $\endgroup$
    – Tal Galili
    Apr 3, 2011 at 22:16
  • $\begingroup$ p.s: I now see that there was an edit to the title of my question that might had led you to think I was asking about predict.lm interval parameter (which I am not) :) $\endgroup$
    – Tal Galili
    Apr 3, 2011 at 22:18
  • 8
    $\begingroup$ You are abusing terminology here. Excel is not statistical software. $\endgroup$
    – Dirk Eddelbuettel
    Apr 3, 2011 at 22:23
  • 1
    $\begingroup$ You're right, my bid, how about "a spreadsheet application" ? $\endgroup$
    – Tal Galili
    Apr 3, 2011 at 22:42
  • 3
    $\begingroup$ I can live with that; it calls the devil by its name ;-) $\endgroup$
    – Dirk Eddelbuettel
    Apr 3, 2011 at 22:52
7
$\begingroup$

@Tal: Might I suggest Kutner et al as a fabulous source for linear models.

There is the distinction between

  1. a prediction of $Y$ from an individual new observation $X_{vec}$,
  2. the expected value of a $Y$ conditioned on $X_{vec}$, $E(Y|X_{vec})$ and
  3. $Y$ from several instances of $x_{vec}$

They are all covered in detail in the text.

I think you are looking for the formula for the confidence interval around $E(Y|X_{vec})$ and that is $\hat{Y} \pm t_{1-\alpha/2}s_{\hat{Y}}$ where $t$ has $n-2$ d.f. and $s_{\hat{Y}}$ is the standard error of $\hat{Y}$, $\frac{\sigma^{2}}{n} +(X_{vec}-\bar{X})^{2}\frac{\sigma^{2}}{\sum(X_{i}-\bar{X})^{2}}$

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ (+1) for making the distinction. However, I believe the OP is asking for (1), not (2) (and I have edited the question's title accordingly). Also note that your formula appears to assume the regression depends only on one variable. $\endgroup$
    – whuber ♦
    Apr 4, 2011 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.