18
$\begingroup$

I understand the concept that $\hat\beta_0$ is the mean for when the categorical variable is equal to 0 (or is the reference group), giving the end interpretation that the regression coefficient is the difference in mean of the two categories. Even with >2 categories I would assume each $\hat\beta$ explains the difference between that category's mean and the reference.

But, what if more variables are brought into the multivariable model? Now what does the intercept mean given that it doesn't make sense for it to be the mean for the reference of two categorical variables? An example would be if gender (M(ref)/F) and race (white(ref)/black) were both in a model. Is the $\hat\beta_0$ the mean for only white males? How does one interpret any other possibilities?

As a separate note: do contrast statements serve as a way to method for investigating effect modification? Or just to see the effect ($\hat\beta$) at different levels?

$\endgroup$
  • $\begingroup$ As a terminological note, "multivariate" means multiple response variables, not multiple predictor variables (see here). Also, I don't follow your last question. $\endgroup$ – gung - Reinstate Monica Oct 14 '14 at 19:31
  • $\begingroup$ Thanks for this clarification. Getting the language correct is important to me! I suppose I just can't figure out why contrast statements are used at all since one could always just set the reference variable to that which one is contrasting against? $\endgroup$ – Renee Oct 15 '14 at 18:41
  • 1
    $\begingroup$ I guess you could just keep re-fitting the model w/ different reference levels. I'm not sure that's more convenient. With contrasts, you can also specify a set of orthogonal contrasts or a theoretically implied contrast (A vs combination of B&C) to test. $\endgroup$ – gung - Reinstate Monica Oct 15 '14 at 20:37
19
$\begingroup$

You are right about the interpretation of the betas when there is a single categorical variable with $k$ levels. If there were multiple categorical variables (and there were no interaction term), the intercept ($\hat\beta_0$) is the mean of the group that constitutes the reference level for both (all) categorical variables. Using your example scenario, consider the case where there is no interaction, then the betas are:

  • $\hat\beta_0$: the mean of white males
  • $\hat\beta_{\rm Female}$: the difference between the mean of females and the mean of males
  • $\hat\beta_{\rm Black}$: the difference between the mean of blacks and the mean of whites

We can also think of this in terms of how to calculate the various group means:
\begin{align} &\bar x_{\rm White\ Males}& &= \hat\beta_0 \\ &\bar x_{\rm White\ Females}& &= \hat\beta_0 + \hat\beta_{\rm Female} \\ &\bar x_{\rm Black\ Males}& &= \hat\beta_0 + \hat\beta_{\rm Black} \\ &\bar x_{\rm Black\ Females}& &= \hat\beta_0 + \hat\beta_{\rm Female} + \hat\beta_{\rm Black} \end{align}

If you had an interaction term, it would be added at the end of the equation for black females. (The interpretation of such an interaction term is quite convoluted, but I walk through it here: Interpretation of interaction term.)


Update: To clarify my points, let's consider a canned example, coded in R.

d = data.frame(Sex  =factor(rep(c("Male","Female"),times=2), levels=c("Male","Female")),
               Race =factor(rep(c("White","Black"),each=2),  levels=c("White","Black")),
               y    =c(1, 3, 5, 7))
d
#      Sex  Race y
# 1   Male White 1
# 2 Female White 3
# 3   Male Black 5
# 4 Female Black 7

enter image description here

The means of y for these categorical variables are:

aggregate(y~Sex,  d, mean)
#      Sex y
# 1   Male 3
# 2 Female 5
## i.e., the difference is 2
aggregate(y~Race, d, mean)
#    Race y
# 1 White 2
# 2 Black 6
## i.e., the difference is 4

We can compare the differences between these means to the coefficients from a fitted model:

summary(lm(y~Sex+Race, d))
# ...
# Coefficients:
#             Estimate Std. Error  t value Pr(>|t|)    
# (Intercept)        1   3.85e-16 2.60e+15  2.4e-16 ***
# SexFemale          2   4.44e-16 4.50e+15  < 2e-16 ***
# RaceBlack          4   4.44e-16 9.01e+15  < 2e-16 ***
# ...
# Warning message:
#   In summary.lm(lm(y ~ Sex + Race, d)) :
#   essentially perfect fit: summary may be unreliable

The thing to recognize about this situation is that, without an interaction term, we are assuming parallel lines. Thus, the Estimate for the (Intercept) is the mean of white males. The Estimate for SexFemale is the difference between the mean of females and the mean of males. The Estimate for RaceBlack is the difference between the mean of blacks and the mean of whites. Again, because a model without an interaction term assumes that the effects are strictly additive (the lines are strictly parallel), the mean of black females is then the mean of white males plus the difference between the mean of females and the mean of males plus the difference between the mean of blacks and the mean of whites.

$\endgroup$
  • $\begingroup$ Thank you! Very clear & helpful. At the end you mention interaction terms. If one does an interaction term then how does this alter the betas (meaning the new betas from the interaction term model)? I know that the p value for the interaction term is important, but does the interaction term beta have a meaningful interpretation? Thank you again for your help! $\endgroup$ – Renee Oct 15 '14 at 18:35
  • 1
    $\begingroup$ In the case of an interaction, the 'main effect' betas only refer to the differences within the reference level of the other factor. Eg, $\hat\beta_{\rm Female}$ is only the difference between $\bar x_{\rm White\ Male}$ and $\bar x_{\rm White\ Female}$. $\endgroup$ – gung - Reinstate Monica Oct 15 '14 at 20:40
  • $\begingroup$ Makes sense. Thank you! & it is altered from the model without interaction term due to the interaction term ameliorating the main effect? Meaning if there is no interaction the main effect term would theoretically be the same? $\endgroup$ – Renee Oct 15 '14 at 21:29
  • $\begingroup$ If the interaction effect were exactly 0 (to infinite decimal places), not only in the population, but also in your sample, the main effect betas would be the same in a model w/ or w/o the interaction term. $\endgroup$ – gung - Reinstate Monica Oct 15 '14 at 21:32
  • 1
    $\begingroup$ @hans0l0, that would be better as a new question rather than information buried here in comments; you could link to this for context. Briefly, it is the mean of the reference levels when all continuous variables are =0. $\endgroup$ – gung - Reinstate Monica Oct 10 '15 at 19:08
6
$\begingroup$

Actually as you correctly pointed out, in the case of a single categorical variable (with potentially more than 2 levels), $\hat{\beta}_0$ is indeed the mean of the reference and the other $\hat\beta$ are the difference between the mean of that level of the category and the mean of the reference.

If we extend a bit your example to include a third level to the race category (say Asian) and chose White as the reference, then you would have:

  • $\hat{\beta}_0 = \bar{x}_{White}$
  • $\hat{\beta}_{Black} = \bar{x}_{Black} - \bar{x}_{White}$
  • $\hat{\beta}_{Asian} = \bar{x}_{Asian} - \bar{x}_{White}$

In this case, the interpretation of all the $\hat{\beta}$ is easy and finding the mean of any level of the category is straightforward. For example:

  • $\bar{x}_{Asian} = \hat{\beta}_{Asian} + \hat{\beta}_0$

Unfortunately in the case of multiple categorical variables, the correct interpretation for the intercept is no longer as clear (see note at the end). When there is n categories, each with multiple levels and one reference level (e.g. White and Male in you example), the general form for the intercept is:

$$\hat{\beta}_0 =∑_{i=1}^{n}\bar{x}_{reference,i} -(n-1) \bar{x} ,$$ where $$\bar{x}_{reference,i}\small{\text{ is the mean of the reference level of the i-th categorical variable,}}$$ $$\bar{x}\small{\text{ is the mean of the whole data set}}$$

The other $\hat\beta$ are the same as with a single category: they are the difference between the mean of that level of the category and the mean of the reference level of the same category.

If we go back to your example, we would get:

  • $\hat{\beta}_0 = \bar{x}_{White} + \bar{x}_{Male} - \bar{x}$
  • $\hat{\beta}_{Black} = \bar{x}_{Black} - \bar{x}_{White}$
  • $\hat{\beta}_{Asian} = \bar{x}_{Asian} - \bar{x}_{White}$
  • $\hat{\beta}_{Female} = \bar{x}_{Female} - \bar{x}_{Male}$

You will notice that the mean of the cross categories (e.g. White males) are not present in any of the $\hat\beta$. As a matter of fact, you cannot calculate these means precisely from the results of this type of regression.

The reason for this is that, the number of predictor variables (i.e. the $\hat\beta$) is smaller then the number of cross categories (as long as you have more than 1 category) so a perfect fit is not always possible. If we go back to your example, the number of predictors is 4 (i.e. $\hat{\beta}_0, ~\hat{\beta}_{Black}, ~\hat{\beta}_{Asian}$ and $\hat{\beta}_{Female}$) while the number of cross categories is 6.

Numerical Example

Let me borrow from @Gung for a canned numerical example:

d = data.frame(Sex=factor(rep(c("Male","Female"),times=3), levels=c("Male","Female")),
    Race =factor(rep(c("White","Black","Asian"),each=2),levels=c("White","Black","Asian")),
    y    =c(0, 3, 7, 8, 9, 10))
d

#      Sex  Race  y
# 1   Male White  0
# 2 Female White  3
# 3   Male Black  7
# 4 Female Black  8
# 5   Male Asian  9
# 6 Female Asian 10

In this case, the various averages that will go in the calculation of the $\hat\beta$ are:

aggregate(y~1,  d, mean)

#          y
# 1 6.166667

aggregate(y~Sex,  d, mean)

#      Sex        y
# 1   Male 5.333333
# 2 Female 7.000000

aggregate(y~Race, d, mean)

#    Race   y
# 1 White 1.5
# 2 Black 7.5
# 3 Asian 9.5

We can compare these numbers with the results of the regression:

summary(lm(y~Sex+Race, d))

# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)
# (Intercept)   0.6667     0.6667   1.000   0.4226
# SexFemale     1.6667     0.6667   2.500   0.1296
# RaceBlack     6.0000     0.8165   7.348   0.0180
# RaceAsian     8.0000     0.8165   9.798   0.0103

As you can see, the various $\hat\beta$ estimated from the regression all line up with the formulas given above. For example, $\hat\beta_0$ is given by: $$\hat{\beta}_0 = \bar{x}_{White} + \bar{x}_{Male} - \bar{x}$$ Which gives:

1.5 + 5.333333 - 6.166667
# 0.66666

Note on the choice of contrast

A final note on this topic, all the results discussed above relate to categorical regressions using contrast treatment (the default type of contrast in R). There are different types of contrast which could be used (notably Helmert and sum) and and it would change the interpretation of the various $\hat\beta$. However, It would not change the final predictions from the regressions (e.g. the prediction for White males is always the same no matter which type of contrast you use).

My personal favourite is contrast sum as I feel that the interpretation of the $\hat\beta^{contr.sum}$ generalises better when there are multiple categories. For this type of contrast, there is no reference level, or rather the reference is the mean of the whole sample, and you have the following $\hat\beta^{contr.sum}$:

  • $\hat\beta_0^{contr.sum}=\bar{x}$
  • $\hat\beta_i^{contr.sum}=\bar{x}_i-\bar{x}$

If we go back to the previous example, you would have:

  • $\hat{\beta}_0^{contr.sum} = \bar{x}$
  • $\hat{\beta}_{White}^{contr.sum} = \bar{x}_{White} - \bar{x}$
  • $\hat{\beta}_{Black}^{contr.sum} = \bar{x}_{Black} - \bar{x}$
  • $\hat{\beta}_{Asian}^{contr.sum} = \bar{x}_{Asian} - \bar{x}$
  • $\hat{\beta}_{Male}^{contr.sum} = \bar{x}_{Male} - \bar{x}$
  • $\hat{\beta}_{Female}^{contr.sum} = \bar{x}_{Female} - \bar{x}$

You will notice that because White and Male are no longer reference levels, their $\hat\beta^{contr.sum}$ are no longer 0. The fact that these are 0 is specific to contrast treatment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.