Interpretation of predicted values in model comparisons (regression)

Question

I am interested in understanding how to interpret the difference in predicted values across two partially related linear multiple regression models.

Let's assume that the first model is (in Wilkinson notation) " $ y_{full} \approx 1 + x_1 + x_2 + x_3 $ ", so that the variable $y$ is modeled using the "full set" of regressors available. We call this the Full model. ("1" is the intercept)

Let's assume that the second model is " $ y_{partial} \approx 1 + x_2 + x_3 $ " , so that the variable $y$ is modeled using the "full set" of regressors available, excluding $x_1$. We call this the Partial model.

Now, let's pretend that we have some new values of the independent variables ( $ \tilde x_1, \tilde x_2, \tilde x_3 $ ) and use these values to predict y. Of course, all the independent variables are used in the full model, while the partial model only uses the last two, exluding $x_1$. We would obtain $ \hat y_{Full}$ and $ \hat y_{Partial} $, that is, the two predicted values (or the two predicted vectors).

I am interested in the meaning of the difference between the two predicted values ($\Delta$). Mathematically, given $\beta$ as the slopes and $\alpha$ as the intercepts, such difference can be formalised as:

$$ \Delta_{Full,Partial} = \hat y_{Full} - \hat y_{Partial} = \tilde x_{2} ( \beta_{2,Full} - \beta_{2,Partial} ) + \tilde x_{3} ( \beta_{3,Full} - \beta_{3,Partial} ) + \tilde x_{1} (\beta_{1,Full}) + \alpha_{Full} - \alpha_{Partial} $$

(note that $\alpha$ = intercept)

However, I would like to understand the practical meaning of such difference. For example, it is good to say that $\Delta$ indicates the "contribution of $x_1$ to $y$ given the existence/effect of $x_2$ and $x_3$"?

I hope that the question is clear. I am very curious about how to interpret such difference. If you also have example of applications in experimental studies, I would like to read them.

Thanks in advance.

Answer 1

We are comparing the $Y$-values predicted by the full and partial models for a specific data point that has a specific set of values for the predictors $X_1$, $X_2$, and $X_3$: For this data point the difference between the predicted values is the difference between the specific contribution of $X_1$ to the expected $Y$-value for that data point ($x_{1} \hat{\beta}_1$), and the average contribution of $X_1$ to the expected $Y$-values across all data points ($\bar{x}_1\hat{\beta}_1$). This means that predictions from the full and partial models will be quite similar for those data points where the value of $X_1$ is close to its average. The further $X_1$ is from its average, the larger the difference between the two predictions.

At least the above statement is true if we assume that the omitted variable, $X_1$, has zero correlation¹ to the included variables, $X_2$ and $X_3$. In this case the estimates of the slope parameters $\beta_2$ and $\beta_3$ will be identical for the partial and full models². However, the estimates of the intercept parameter, $\alpha$, will differ. Specifically the $\alpha$ estimate for the partial model will include the average effect of the omitted variable (in the equations below I use the subscript "Partial" for parameters from the partial model, and no subscript for parameters from the full model):

$$ \begin{aligned} &\hat{\alpha}_{Partial} = \hat{\alpha} + \bar{x}_1\hat{\beta}_{1}\\ &\hat{\beta}_{2,Partial} = \hat{\beta}_{2}\\ &\hat{\beta}_{3,Partial} = \hat{\beta}_{3} \end{aligned} $$

Here $\bar{x}_1$ is the mean of the variable $X_1$ in the sample that is being examined. Given these relationships we can derive an expression for the difference between the predicted values for the full and partial models:

$$ \begin{aligned} \Delta_{Full,Partial} &= \hat{y} - \hat{y}_{Partial}\\ &= \hat{\alpha} + x_{1} \hat{\beta}_{1} + x_{2} \hat{\beta}_{2} + x_{3} \hat{\beta}_{3} - \hat{\alpha}_{Partial} - x_{2} \hat{\beta}_{2,Partial} - x_{3} \hat{\beta}_{3,Partial}\\ &= \hat{\alpha} + x_{1} \hat{\beta}_{1} + x_{2} \hat{\beta}_{2} + x_{3} \hat{\beta}_{3} - \hat{\alpha} - \bar{x}_1\hat{\beta}_{1} - x_{2} \hat{\beta}_{2} - x_{3} \hat{\beta}_{3}\\ &= x_{1} \hat{\beta}_{1} - \bar{x}_1\hat{\beta}_{1} \end{aligned} $$

Thus, as stated above, the difference in predictions for a given data point is the difference between the specific contribution of $X_1$ to $\hat{Y}$ for that data point, and the average contribution of $X_1$ to $\hat{Y}$ across all data points. Essentially, the partial model "knows" nothing about the specific value of $X_1$ for any given data point, and can therefore not account for the specific contributions of this variable. Instead the partial models adds the average contribution of $X_1$ (across all data) to all predictions. The full model, however, will know exactly how much $X_1$ contributes to the exptected value of $Y$ for each data point.

---

¹Note: this means zero correlation in the specific sample that is being examined. It does not matter if the underlying population parameters are correlated or not. Also note that you are unlikely to have zero sample correlation for real-world data, except in cases where the experimenter has control of the independent variables and their values have been chosen explicitly to be uncorrelated. Nevertheless, looking at the situation with zero correlation is helpful in getting some intuition about the meaning of the difference in predictions.

²For instance, see: Eric A. Hanushek and John E. Jackson, Statistical Methods for Social Scientists, Academic Press, 1977, section 4.3.

Also: The Phantom Menace: Kevin A. Clarke, Omitted Variable Bias in Econometric Research, Conflict Management and Peace Science, 22:341-352, 2005

Answer 2

I’d call $\Delta$ “the change from adding $x_1$ as a predictor”.

As an example, suppose $y$ is a final exam grade, $x_1$ is a midterm grade, $x_2$ is a participation grade from the first half of a class, and $x_3$ is whether the student had taken a related class beforehand, and the regressions come from last semester’s data. Then with new data on this semester’s students, $\Delta$ tells you the change in predictions for the final given the performance on the midterm.

Calling $\Delta$ “the contribution of the midterm grade to the prediction of the final grade” is awkward. You can make the phrase “the contribution of $x_1$” precise with a formula, and there may be areas where that phrase has a well-established meaning, but in general misunderstandings are so likely that I’d avoid talk of statistical “contributions” entirely.

Answer 3

I usually don't look at (nested) model comparison this way, i.e. in the perspective of the difference of predictions. When I have multiple predictors the question in general is whether to add a predictor in the regression or not. In this I follow Gelman's advise (for example, Gelman et al. Data Analysis using Multilevel Regression) to include predictors if they make sense and if they contribute some information. That is if the weight of a predictor is practically zero you may exclude it but if it makes sense to have it in the regression you can leave it and this will change your interpretation of the model.

To address more directly the question I prefer to interpret the difference of coefficients (or actually even better the distribution of the coefficient with and without the new predictor) more than the difference of predictions itself. For example if adding $x_1$ makes the distribution of coefficient $\beta_{2, Full}$ centred in zero with a small variance while the distribution $\beta_{2, Partial}$ has most density on the positive side I would interpret that the effect of $x_2$ indicated by the partial model is not relevant when you take into account $x_1$.

Hope this helps.