4
$\begingroup$

I have a cross-sectional multiple regression that I have estimated and now I would like to apply it to make a simple forecast of the dependent variable.

Take the data generating process

$$y_i =\alpha+\beta x_i+\Gamma Z_i+\epsilon_i,$$

where $y$ is the dependent variable, $x$ the independent variable and $Z$ some vector of covariates. I have performed the regression for the entire sample and estimated $\alpha$, $\beta$ and $\Gamma$ accordingly.

Now, I would like to make an out-of-sample forecast of $y_j$, say, at indicator $j$. All of $Z_j$ is known to me; however, $x_j$ is not. Now, it itches to take the average or conditional average of $x$, i.e., $E[x|..]$, of the original sample and use the it to predict $y_j$. That is

$$\hat y_j =\hat\alpha+\hat\beta \bar x+\hat \Gamma Z_j.$$

Would using the average or conditional improve the forecast in any way?

Intuitively it appears to me the average or conditional average does not store any additional information to improve the prediction per se, but, if I have to leave out $x_i$ from the original regression and re-estimate it and then do a forecast without $x_i$, it appears that I am losing out on the information stored in $x_i$ as well.

$\endgroup$
  • $\begingroup$ I just want to confirm that you are doing multiple regression on independent covariates and not undertaking a data reduction step such as PCA or PLS? $\endgroup$ – ReneBt Mar 6 at 11:11
4
+100
$\begingroup$

Let us write the regression problem as:

$$y_i = \alpha + \beta x_i + \Gamma Z_i + e_i$$

We can rewrite $\beta x_i$ as $\beta \bar{x} + \beta (x_i -\bar{x})$:

$$y_i = \alpha + \beta \bar{x} + \beta (x_i-\bar{x}) + \Gamma Z_i + e_i$$

Since $\bar{x}$ is a constant, the term $\beta \bar{x}$ can be absorbed into the constant term:

$$y_i = \alpha' + \beta (x_i-\bar{x}) + \Gamma Z_i + e_i$$

Note that this is mathematically equivalent to the original regression problem and can be estimated as one; you just get out a different parameter estimate for $\alpha$ than in the original problem, but you can recover it via:

$$\hat{\alpha} = \hat{\alpha}' - \hat{\beta} \bar{x}$$

Your estimates for $\beta$ will be the same in the two regressions, but your estimates of the constant terms won't be, unless $\bar{x}=0$, and this is the key to understanding what happens with your problem.

If we don't know $x_j - \bar{x}$ for some future observations indexed by $j$, and we therefore exclude all terms involving $x_j$ and $\bar{x}$, we are setting $x_j-\bar{x} = 0$ in the third equation, so the term $\beta(x_j-\bar{x})$ disappears, but we are also subtracting the term $\hat{\beta}\bar{x}$ from the estimated constant term $\alpha'$, which does not in fact change with changes in the value of $x_j$. This will cause our prediction of $y_j$ to be biased by an amount equal to $\hat{\beta}\bar{x}$.

Another way to look at is that if you remove the $x_j$ from the model altogether, you'll get the same answer as if you had substituted $0$ for all the $x_j$. Naturally, substituting $0$ for all the $x_j$ is not the same as substituting $\bar{x}$, unless $\bar{x}=0$, and, again, you will have effectively removed the term $\hat{\beta}\bar{x}$ from the prediction equation - even though it was part of the estimated equation.

A concrete example follows. Our true regression equation is $y_i = 1 + x_i + e_i$, with $e_i$ i.i.d. Normal(0,1) and $x_i \sim U(5,15)$, therefore with mean $10$:

> x <- runif(100, 5, 15)
> y <- 1 + x + rnorm(100)
> model <- lm(y~x)
> 
> x_future <- runif(100, 5, 15)
> y_future <- 1 + x_future + rnorm(100)
> 
> # Mean of future observed values of y
> mean(y_future)
[1] 11.05226
> 
> # Mean prediction if we have all the x values available
> mean(predict(model, newdata=data.frame(x_future)))
[1] 10.70136
> 
> # Mean prediction if we substitute the mean x value
> mean(predict(model, 
+              newdata=data.frame(x_future=rep(mean(x_future),100))))
[1] 10.70136
> 
> # Prediction if we remove all the x values (leaving just the intercept)
> coefficients(model)[[1]]
[1] 0.1992937

As we can see, substituting $\bar{x}$ for the $x_j$ preserves the mean of our predictions relative to predictions made with the $x_j$ themselves, but removing $x$ altogether does not.

One caveat, though; your prediction confidence intervals will be too narrow in your situation, because they don't take into account the added variability due to not knowing the true $x_i$ and using $\bar{x}$ instead. In effect, you have another error term - $\beta(x_i-\bar{x})$ - that's not accounted for in the estimation procedure.

$\endgroup$
  • $\begingroup$ Thanks@jbowman, this is a great answer and confirmed my intuition and observations that I have made in bivariate regressions. What about using the unconditional mean vs conditional mean E[xi|Zn..Zk] in a multiple regression? For example,if I know that, xj has some categorical characteristics, would this help to improve the forecast? Intuitively I believe we must use the conditional mean in the first place to keep the regression unbiased? Correct? $\endgroup$ – CodeTrek Mar 3 at 20:11
  • $\begingroup$ You are correct; the closer you can get to the conditional mean $E[x_j|Z_j]$ the more accurate your forecast will be. The ideal case (well, given that you don't know the true values of $x$) is when you do know $E[x_j|Z_j]$, in which case that is what you want to use. $\endgroup$ – jbowman Mar 3 at 20:35
  • $\begingroup$ This makes forecasting a very tedious business indeed. For example, one would like to predict pts in a basketball game with thousands of games previously observed. Obviously it depends on the player's throws in each game, which again depends on their age, ability, opp defence etc. You first estimate the coefs for throws in your original regression. Now, for the forecast, you do not know throws to be, but you are better off to use (conditional) average of past throws instead or leaving throws out of the forecast? There are so many other variables you could do the same. $\endgroup$ – CodeTrek Mar 3 at 21:01
  • $\begingroup$ The way to do that is to build a model of points as a function of past performance averages of the two teams involved, rather than as a function of performance in the same game, which, as you note, then requires you essentially to forecast the detailed player performances in the game. $\endgroup$ – jbowman Mar 4 at 0:40
  • 1
    $\begingroup$ Note that you can also use a distribution $P(x_j)$ or $P(x_j | Z_j)$, take samples from this distribution and compute prediction for each sample. This way you get a range of predictions that reflect your uncertainty about $x_j$. $\endgroup$ – Martin Modrák Mar 5 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.