3
$\begingroup$

I am interested in understanding how to interpret the difference in predicted values across two partially related linear multiple regression models.

Let's assume that the first model is (in Wilkinson notation) " $ y_{full} \approx 1 + x_1 + x_2 + x_3 $ ", so that the variable $y$ is modeled using the "full set" of regressors available. We call this the Full model. ("1" is the intercept)

Let's assume that the second model is " $ y_{partial} \approx 1 + x_2 + x_3 $ " , so that the variable $y$ is modeled using the "full set" of regressors available, excluding $x_1$. We call this the Partial model.

Now, let's pretend that we have some new values of the independent variables ( $ \tilde x_1, \tilde x_2, \tilde x_3 $ ) and use these values to predict y. Of course, all the independent variables are used in the full model, while the partial model only uses the last two, exluding $x_1$. We would obtain $ \hat y_{Full}$ and $ \hat y_{Partial} $, that is, the two predicted values (or the two predicted vectors).

I am interested in the meaning of the difference between the two predicted values ($\Delta$). Mathematically, given $\beta$ as the slopes and $\alpha$ as the intercepts, such difference can be formalised as:

$$ \Delta_{Full,Partial} = \hat y_{Full} - \hat y_{Partial} = \tilde x_{2} ( \beta_{2,Full} - \beta_{2,Partial} ) + \tilde x_{3} ( \beta_{3,Full} - \beta_{3,Partial} ) + \tilde x_{1} (\beta_{1,Full}) + \alpha_{Full} - \alpha_{Partial} $$

(note that $\alpha$ = intercept)

However, I would like to understand the practical meaning of such difference. For example, it is good to say that $\Delta$ indicates the "contribution of $x_1$ to $y$ given the existence/effect of $x_2$ and $x_3$"?

I hope that the question is clear. I am very curious about how to interpret such difference. If you also have example of applications in experimental studies, I would like to read them.

Thanks in advance.

$\endgroup$
4
  • $\begingroup$ I'm not familiar with Wilkinson notation, and I expect others are not either. What does the prime symbol (′) mean in this context? $\endgroup$
    – Eli
    Jul 6, 2021 at 15:31
  • $\begingroup$ Do the $\alpha$ terms refer to the intercept of each model? $\endgroup$
    – Eli
    Jul 6, 2021 at 15:32
  • $\begingroup$ @Eli: I was also confused by the prime, but it appears that it is actually an end quote (note there is an opening quote in front of y_full and y_partial) $\endgroup$ Jul 7, 2021 at 7:36
  • $\begingroup$ @Eli, now the quotes are outside the formula, hope it is more readable. Thanks. $\endgroup$
    – smndpln
    Jul 7, 2021 at 8:39

3 Answers 3

1
$\begingroup$

We are comparing the $Y$-values predicted by the full and partial models for a specific data point that has a specific set of values for the predictors $X_1$, $X_2$, and $X_3$: For this data point the difference between the predicted values is the difference between the specific contribution of $X_1$ to the expected $Y$-value for that data point ($x_{1} \hat{\beta}_1$), and the average contribution of $X_1$ to the expected $Y$-values across all data points ($\bar{x}_1\hat{\beta}_1$). This means that predictions from the full and partial models will be quite similar for those data points where the value of $X_1$ is close to its average. The further $X_1$ is from its average, the larger the difference between the two predictions.

At least the above statement is true if we assume that the omitted variable, $X_1$, has zero correlation1 to the included variables, $X_2$ and $X_3$. In this case the estimates of the slope parameters $\beta_2$ and $\beta_3$ will be identical for the partial and full models2. However, the estimates of the intercept parameter, $\alpha$, will differ. Specifically the $\alpha$ estimate for the partial model will include the average effect of the omitted variable (in the equations below I use the subscript "Partial" for parameters from the partial model, and no subscript for parameters from the full model):

$$ \begin{aligned} &\hat{\alpha}_{Partial} = \hat{\alpha} + \bar{x}_1\hat{\beta}_{1}\\ &\hat{\beta}_{2,Partial} = \hat{\beta}_{2}\\ &\hat{\beta}_{3,Partial} = \hat{\beta}_{3} \end{aligned} $$

Here $\bar{x}_1$ is the mean of the variable $X_1$ in the sample that is being examined. Given these relationships we can derive an expression for the difference between the predicted values for the full and partial models:

$$ \begin{aligned} \Delta_{Full,Partial} &= \hat{y} - \hat{y}_{Partial}\\ &= \hat{\alpha} + x_{1} \hat{\beta}_{1} + x_{2} \hat{\beta}_{2} + x_{3} \hat{\beta}_{3} - \hat{\alpha}_{Partial} - x_{2} \hat{\beta}_{2,Partial} - x_{3} \hat{\beta}_{3,Partial}\\ &= \hat{\alpha} + x_{1} \hat{\beta}_{1} + x_{2} \hat{\beta}_{2} + x_{3} \hat{\beta}_{3} - \hat{\alpha} - \bar{x}_1\hat{\beta}_{1} - x_{2} \hat{\beta}_{2} - x_{3} \hat{\beta}_{3}\\ &= x_{1} \hat{\beta}_{1} - \bar{x}_1\hat{\beta}_{1} \end{aligned} $$

Thus, as stated above, the difference in predictions for a given data point is the difference between the specific contribution of $X_1$ to $\hat{Y}$ for that data point, and the average contribution of $X_1$ to $\hat{Y}$ across all data points. Essentially, the partial model "knows" nothing about the specific value of $X_1$ for any given data point, and can therefore not account for the specific contributions of this variable. Instead the partial models adds the average contribution of $X_1$ (across all data) to all predictions. The full model, however, will know exactly how much $X_1$ contributes to the exptected value of $Y$ for each data point.


1Note: this means zero correlation in the specific sample that is being examined. It does not matter if the underlying population parameters are correlated or not. Also note that you are unlikely to have zero sample correlation for real-world data, except in cases where the experimenter has control of the independent variables and their values have been chosen explicitly to be uncorrelated. Nevertheless, looking at the situation with zero correlation is helpful in getting some intuition about the meaning of the difference in predictions.

2For instance, see: Eric A. Hanushek and John E. Jackson, Statistical Methods for Social Scientists, Academic Press, 1977, section 4.3.

Also: The Phantom Menace: Kevin A. Clarke, Omitted Variable Bias in Econometric Research, Conflict Management and Peace Science, 22:341-352, 2005

$\endgroup$
2
  • $\begingroup$ Thank you for this answer, it extremely useful. So, you quite convinced me that the differences between predicted values is readable as "the difference between the specific contribution of X1 to Y for that data point, and the average contribution of X1 to Y". That makes me think, however, that in case of (non-problematic) non-zero correlations among predictors, the definition becomes something like "the difference between the specific contribution of X1 to Y for that data point, and the average contribution of X1 to Y by accounting for the values in X2 and X3 for that data point". Correct? $\endgroup$
    – smndpln
    Jul 14, 2021 at 14:32
  • $\begingroup$ (I am sorry that the bounty is no more eligible, you earnt it) $\endgroup$
    – smndpln
    Jul 14, 2021 at 14:32
0
$\begingroup$

I’d call $\Delta$ “the change from adding $x_1$ as a predictor”.

As an example, suppose $y$ is a final exam grade, $x_1$ is a midterm grade, $x_2$ is a participation grade from the first half of a class, and $x_3$ is whether the student had taken a related class beforehand, and the regressions come from last semester’s data. Then with new data on this semester’s students, $\Delta$ tells you the change in predictions for the final given the performance on the midterm.

Calling $\Delta$ “the contribution of the midterm grade to the prediction of the final grade” is awkward. You can make the phrase “the contribution of $x_1$” precise with a formula, and there may be areas where that phrase has a well-established meaning, but in general misunderstandings are so likely that I’d avoid talk of statistical “contributions” entirely.

$\endgroup$
3
  • $\begingroup$ Thanks for your answer. I strongly agree with the fact that "contribution" is an ambiguous term, here. Anyway, I find your example a bit misleading, or at least I am not getting it. I was not supposing new data to be added, my interest is only in the intuition about how to interpret the Delta. $\endgroup$
    – smndpln
    Jul 7, 2021 at 8:41
  • $\begingroup$ I considered the new values of the independent variables to be new data — did I miss some distinction? $\endgroup$
    – Matt F.
    Jul 7, 2021 at 10:10
  • $\begingroup$ I am not sure. My point is that having new data or not is irrelevant for the question. $\endgroup$
    – smndpln
    Jul 7, 2021 at 12:22
0
$\begingroup$

I usually don't look at (nested) model comparison this way, i.e. in the perspective of the difference of predictions. When I have multiple predictors the question in general is whether to add a predictor in the regression or not. In this I follow Gelman's advise (for example, Gelman et al. Data Analysis using Multilevel Regression) to include predictors if they make sense and if they contribute some information. That is if the weight of a predictor is practically zero you may exclude it but if it makes sense to have it in the regression you can leave it and this will change your interpretation of the model.

To address more directly the question I prefer to interpret the difference of coefficients (or actually even better the distribution of the coefficient with and without the new predictor) more than the difference of predictions itself. For example if adding $x_1$ makes the distribution of coefficient $\beta_{2, Full}$ centred in zero with a small variance while the distribution $\beta_{2, Partial}$ has most density on the positive side I would interpret that the effect of $x_2$ indicated by the partial model is not relevant when you take into account $x_1$.

Hope this helps.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the answer! Usually I don't look at models in this way, either. That's why I asked the question =). I partially agree on the fact that interpreting coefficients (or differences in coefficients) may be more interpretable. However, my question is about the interpretability of Delta(full, partial) for predicted values. It feels strange to me that such "simple" difference is so difficult to describe practically. $\endgroup$
    – smndpln
    Jul 8, 2021 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.