10
$\begingroup$

I'm failing to understand the value of the intercept value in a multiple linear regression with categorical values. Taking the "warpbreaks" data set as an example, when I do:

> lm(breaks ~ wool, data=warpbreaks)

Call:
lm(formula = breaks ~ wool, data = warpbreaks)

Coefficients:
(Intercept)        woolB
     31.037       -5.778

I'm able to understand that the value of intercept is the mean value of breaks when wool equals "A", and that adding up the "woolB" coefficient to the intercept value I get the mean value of breaks when wool equals "B". However, if I also consider the tension variable in the model, I'm unable to figure out the meaning of the intercept value:

> lm(breaks ~ wool + tension, data=warpbreaks)

Call:
lm(formula = breaks ~ wool + tension, data = warpbreaks)

Coefficients:
(Intercept)        woolB     tensionM     tensionH
     39.278       -5.778      -10.000      -14.722

I thought it would be the mean value of breaks when either wool equals "A" or tension equals "L", but that isn't true for this dataset.

Any clues on interpreting the value of intercept?

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ It would be the mean when wool=="A" AND tension=="L" if you also included the interaction between wool and tension in the model. Without that, I think you're just left with an estimate of the mean, assuming the two factors have independent effects. $\endgroup$
    – Macro
    Commented Jul 27, 2012 at 12:13
  • $\begingroup$ Assuming the factors have independent effects, do you know how that estimate is produced? $\endgroup$
    – jcazevedo
    Commented Jul 27, 2012 at 12:21
  • 3
    $\begingroup$ The model you've written is $Y_i = \beta_0 + \beta_1 B + \beta_2 M + \beta_3 H + \varepsilon_i$ where $B,M,H$ denote dummy variables for the levels of the variables in question. The $\beta$ estimates are chosen to minimize the residual sums of squares - is that what you're asking? $\endgroup$
    – Macro
    Commented Jul 27, 2012 at 12:23

1 Answer 1

Reset to default
14
$\begingroup$

Contrary to intuition, this is not the mean value of breaks when wool=="A" and tension=="L".

data(warpbreaks)
aggregate(breaks ~ wool + tension, warpbreaks, mean)
#   wool tension   breaks
# 1    A       L 44.55556
# 2    B       L 28.22222
# 3    A       M 24.00000
# 4    B       M 28.77778
# 5    A       H 24.55556
# 6    B       H 18.77778

As @Macro explains in his comments, this depends very much on the model you fit. If you fit the full model (with interaction terms) you get the following:

lm(breaks ~ wool * tension, data=warpbreaks)
#
# Call:
# lm(formula = breaks ~ wool * tension, data = warpbreaks)
# 
# Coefficients:
#    (Intercept)           woolB        tensionM        tensionH woolB:tensionM
#          44.56          -16.33          -20.56          -20.00           21.11
# woolB:tensionH  
#          10.56

where now the intercept is the mean values of breaks when wool=="A" and tension=="L".

This is so because in the full model, there is one parameter per case (6 parameters in total as you can check), while in the additive model there are less parameters than cases (4 parameters in total).

Even though the intercept is not the mean value, notice that the difference between the mean values of breaks when wool=="B" and when wool=="A" is equal to the parameter woolB

aggregate(breaks ~ wool, data=warpbreaks, mean)
#   wool   breaks
# 1    A 31.03704
# 2    B 25.25926
25.25926 - 31.03704
# [1] -5.77778

Likewise, you can check that the same holds true for tension.

aggregate(breaks ~ tension, data=warpbreaks, mean)
#   tension   breaks
# 1       L 36.38889
# 2       M 26.38889
# 3       H 21.66667
26.38889 - 36.38889
# [1] -10
21.66667 - 36.38889
# [1] -14.72222

In conclusion, when you fit an additive model (no interaction term), the parameters are the difference of the mean per category (of only one factor) and the intercept is the estimated value of the response variable for the first modalities of each factor under the assumption of additivity.

This estimate may not be reasonable, if additivity does not hold. You can get an idea whether this assumption is reasonable by testing the nullity of interaction terms.

anova(lm(breaks ~ wool*tension, data=warpbreaks))
# Analysis of Variance Table
# 
# Response: breaks
#              Df Sum Sq Mean Sq F value    Pr(>F)
# wool          1  450.7  450.67  3.7653 0.0582130 .
# tension       2 2034.3 1017.13  8.4980 0.0006926 ***
# wool:tension  2 1002.8  501.39  4.1891 0.0210442 *
# Residuals    48 5745.1  119.69
# ---
# Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

As you can see, the p-value of the test is 0.021, which means that interaction terms can probably not be neglected and that the intercept estimate of the additive model is perhaps not meaningful.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the nice explanation. I have one question how do you calculate here the woolB:tensionM and woolB:tensionH from the aggregate output here? $\endgroup$
    – user134432
    Commented Oct 12, 2016 at 11:26
  • $\begingroup$ @Boby I think you cannot. If you want to know the values, you will have to remove the call to anova(). $\endgroup$
    – gui11aume
    Commented Oct 12, 2016 at 20:23
  • $\begingroup$ Hi @gui11aume. Can you please explain In the additive model, the intercept is the estimated value of the response variable for the first modalities of each factor under the assumption of additivity ? Is it the mean of the response variable in the additive model? $\endgroup$
    – Marouen
    Commented Dec 23, 2019 at 22:34
  • 2
    $\begingroup$ @Marouen no it is not. If the predictors are correlated and you fit an additive model without interaction term, the intercept may be different from the mean response in this category. $\endgroup$
    – gui11aume
    Commented Dec 30, 2019 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.