5
$\begingroup$

I'm comparing two regression models in R. Both use the same categorical variables the only difference is that in one I allow them to interact, the other I don't.

LangFamilyTaskModel <- lm(WER ~ lang_family+Task)
lang_family_task_model <- lm(WER ~ lang_family*Task)

As far as I know the intercept of both of these models should represent the mean of the reference group of both variables (ORF and English). This is what I'm getting though:

───────────────────────────────────────────────────────────────────
                                           Model 1      Model 2    
                                        ───────────────────────────
  (Intercept)                              13.89 ***    16.34 ***  
                                           (1.03)       (1.20)     
  lang_familyOther Indo-European            3.57 *      -2.99      
                                           (1.45)       (2.49)     
  lang_familyNon-Indo-European              7.29 ***     1.82      
                                           (1.31)       (2.25)     
  TaskPD                                    4.73 ***     1.33      
                                           (1.31)       (1.70)     
  TaskSR                                   12.36 ***     8.43 ***  
                                           (1.31)       (1.70)     
  lang_familyOther Indo-European:TaskPD                  9.86 **   
                                                        (3.52)     
  lang_familyNon-Indo-European:TaskPD                    7.07 *    
                                                        (3.18)     
  lang_familyOther Indo-European:TaskSR                  9.84 **   
                                                        (3.52)     
  lang_familyNon-Indo-European:TaskSR                    9.35 **   
                                                        (3.18)     
                                        ───────────────────────────
  N                                         642          642         
  R2                                        0.16         0.18      
───────────────────────────────────────────────────────────────────
  *** p < 0.001; ** p < 0.01; * p < 0.05.                          

The mean of the reference group of both variables is 16.34, so the intercept of the interaction model is what I would expect. But I don't understand what the intercept in Model 1 represents.

I'll show the means and variables here in case it's relevant

  Task         lang_family   N      WER        sd        se       ci
1  ORF             English 126 16.33772 12.445694 1.1087506 2.194355
2  ORF Other Indo-European  38 13.34509  4.807945 0.7799517 1.580332
3  ORF   Non-Indo-European  50 18.15471 11.543049 1.6324337 3.280498
4   PD             English 126 17.66376 10.346593 0.9217477 1.824253
5   PD Other Indo-European  38 24.52889 16.949097 2.7495066 5.571029
6   PD   Non-Indo-European  50 26.55025 17.367827 2.4561817 4.935882
7   SR             English 126 24.76683 12.849972 1.1447665 2.265635
8   SR Other Indo-European  38 31.61183 16.722969 2.7128238 5.496703
9   SR   Non-Indo-European  50 35.93723 19.257220 2.7233821 5.472841
$\endgroup$
  • $\begingroup$ I don't understand the confusion, the second model is saturated (meaning you have included all interaction's effects) using the coefficients in model 2 you can calculate all the mean values found in the table at the button. In the first model you do not allow task and language to depend on eachother and the reference group is therefore not task ORF AND language English, but rather the pooled version of English or ORF $\endgroup$ – Repmat Sep 15 '19 at 19:18
  • $\begingroup$ Thank you for your response @Repmat. I guess maybe this is an obvious question so forgive me, but I still don't know what you mean by the 'pooled version of English or ORF'. Can you explain that a bit further? $\endgroup$ – liam Sep 15 '19 at 20:39
  • $\begingroup$ The intercept (in either model) is calculated as $\hat{\beta}_0 = \bar{y} - \sum_{i=1}^{k} \bar{x}_i \cdot \hat{\beta}_i$ in model 1 the reference group (i.e. the intercept) does not corrospond to any group in the table - it is simply some conflated group of ORF task'ers OR english speakers, and likely its not very interesting $\endgroup$ – Repmat Sep 16 '19 at 7:05
  • $\begingroup$ Thanks @Repmat, I have just edited my answer to apply the formula you have provided to the fake dataset. $\endgroup$ – simone Sep 16 '19 at 9:04
1
$\begingroup$

I think your question is interesting. Actually I don't think @Sympa has answered why the intercept is different between the models with and without the interaction term. A spoiler: I do not have a clear answer to that question but I'll try to do my best to help out. Updated spoiler: at the end it seems a real answer showed up.

The function lm() uses, by-default, the dummy coding (have a look at the contrasts argument). You can find a good explanation of the different codings for categorical predictors in R here.

For the simplest case, when you have a single categorical predictor X your intercept definitively represents the average value of the response variable for X=0, that is for X equal to the reference (baseline) level. R, by default, orders the level of your categorical variables alphabetically (e.g. if sex = male and female, female is the reference level). You can check the levels of your categorical variables using the function levels() and change their order using the function relevel().

A reproducible example is always welcome and I have just created one below that mimics your data structure.

set.seed(1234567895)
lang_family<- factor(sample(3,642,replace=T))
levels(lang_family)[1]<- "English"
levels(lang_family)[2]<- "Other Indo-European"
levels(lang_family)[3]<- "Non-Indo-European"

Task<- factor(sample(3,642,replace=T))
levels(Task)[1]<- "ORF"
levels(Task)[2]<- "PD"
levels(Task)[3]<- "SR"

WER<- rnorm(642,16)

# The variables are random, they do not look like yours except the names of the variables and of the categorical variables' levels.

table(Task,lang_family)# just to have a look
     lang_family
Task  English Other Indo-European Non-Indo-European
  ORF      79                  79                70
  PD       67                  60                82
  SR       75                  57                73

LangFamilyTaskModel<-lm(WER~lang_family+Task)
lang_family_task_model<-lm(WER~lang_family*Task)

summary(LangFamilyTaskModel)

Call:
lm(formula = WER ~ lang_family + Task)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.67896 -0.61602 -0.01906  0.66701  2.82909 

Coefficients:
                               Estimate Std. Error t value Pr(>|t|)    
(Intercept)                    16.05157    0.08225 195.149   <2e-16 ***
lang_familyOther Indo-European -0.03680    0.09421  -0.391    0.696    
lang_familyNon-Indo-European   -0.11338    0.09098  -1.246    0.213    
TaskPD                         -0.01465    0.09212  -0.159    0.874    
TaskSR                          0.02001    0.09250   0.216    0.829    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9592 on 637 degrees of freedom
Multiple R-squared:  0.002796,  Adjusted R-squared:  -0.003466 
F-statistic: 0.4465 on 4 and 637 DF,  p-value: 0.775

summary(lang_family_task_model)

Call:
lm(formula = WER ~ lang_family * Task)

Residuals:
     Min       1Q   Median       3Q      Max 
-2.59478 -0.60578 -0.02055  0.66988  2.80802 

Coefficients:
                                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)                           15.98321    0.10802 147.959   <2e-16 ***
lang_familyOther Indo-European         0.01548    0.15277   0.101    0.919    
lang_familyNon-Indo-European           0.05027    0.15760   0.319    0.750    
TaskPD                                 0.07478    0.15946   0.469    0.639    
TaskSR                                 0.14155    0.15479   0.914    0.361    
lang_familyOther Indo-European:TaskPD -0.11581    0.22905  -0.506    0.613    
lang_familyNon-Indo-European:TaskPD   -0.17086    0.22325  -0.765    0.444    
lang_familyOther Indo-European:TaskSR -0.03848    0.22760  -0.169    0.866    
lang_familyNon-Indo-European:TaskSR   -0.32376    0.22307  -1.451    0.147    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.9601 on 633 degrees of freedom
Multiple R-squared:  0.007018,  Adjusted R-squared:  -0.005531 
F-statistic: 0.5593 on 8 and 633 DF,  p-value: 0.8115


tapply(X=WER,INDEX=list(lang_family,Task),FUN=mean)
                         ORF       PD       SR
English             15.98321 16.05799 16.12476
Other Indo-European 15.99869 15.95766 16.10176
Non-Indo-European   16.03348 15.93740 15.85127

You can see that, as in your case, the intercept is the value of the response variable when both the predictors are at their reference level (as I would have expected) only for the model containing the interaction but not for the model without the interaction. I would really appreciate if someone chime in with a simple and understandable explanation of that. I have found a somehow similar question here but the answers do not dispel any doubts. After reading that post and googling here and there, I am pretty sure the answer has to do with the model being saturated (containing all possible main effects and all possible interaction effects) or not. The interecept is what one would expect for the saturated model but not for models that are not saturated. A simple, conceptual, explanation of why that would be great...

I have edited this answer after the @Repmat comment (thanks for that!), sometimes what is obvious to one is not to others...by the way it was not obvious to me.

So, I have learnt a new and very useful thing today that is how the estimate of the intercept is calculated in a model (with dummy coding). This explains why in a non-saturated model the intercept is not what I thought to be and that one must be very careful with its interpretation, actually it should not be interpreted! Obviously I am not a mathematician and I like to see with my eyes how things look like.

Therefore, applying the equation provided in the @Repmat comment to the fake dataset of my previous answer:

# This is the mean of the response variable:
mean.WER<- mean(data$WER)

# these are is the mean of the dummy-coded categorical variables:
mean.lang_familyOtherIndo_European<- mean(model.matrix(LangFamilyTaskModel)[,2])
mean.lang_familyNon_Indo_European<- mean(model.matrix(LangFamilyTaskModel)[,3])
mean.TaskPD<- mean(model.matrix(LangFamilyTaskModel)[,4])
mean.TaskSR<- mean(model.matrix(LangFamilyTaskModel)[,5])

# these are the betas in the model:
b1<- coef(LangFamilyTaskModel)[2]
b2<- coef(LangFamilyTaskModel)[3]
b3<- coef(LangFamilyTaskModel)[4]
b4<- coef(LangFamilyTaskModel)[5]

# This is applying the equation provided by @Repmat for the non-saturated model
#(it works also for the saturated model):
Int.LangFamilyTaskModel<- mean.WER-sum(b1*mean.lang_familyOtherIndo_European,
+                                        b2*mean.lang_familyNon_Indo_European,
+                                        b3*mean.TaskPD,
+                                        b4*mean.TaskSR)

Int.LangFamilyTaskModel
[1] 15.93446
coef(LangFamilyTaskModel)[1]
(Intercept) 
   15.93446

# they match each other.
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I really don't care about up- and down-votes. However, in this case I am really curious to understand why someone has thought it is a bad answer, just to learn something. I have provided a reproducible example and shift the focus on the issue of saturated vs. non-saturated model that indeed seems to be relevant. What's wrong with my answer? $\endgroup$ – simone Sep 16 '19 at 8:21
  • $\begingroup$ No its a really thorough answer and has helped me a lot even though you are also not entirely sure here. I don't know why it'd get down-voted. $\endgroup$ – liam Sep 16 '19 at 13:49
-1
$\begingroup$

You have developed an interesting Social Science model. And, I don't understand entirely what it means. But, I can clarify a couple of things.

First, in Regression the Intercept is almost never exactly equal to the Mean of your variable. If it is, it is purely accidental. Typically, the more influential your independent variables the more the Intercept will be different from the Mean of the dependent variable. That is not an absolute rule. It is probably wrong in some cases. But, I think in the majority of the cases it is directionally correct.

Second, whenever you add variables to a model all the regression coefficients will typically change. And, that is true when you add interaction variables. When you add such interaction variables, you typically reduce the influence of the original variables by capturing away some of their explanatory power with the interaction variables. And, that is exactly what you are getting here. Within your Model 2, all the absolute values of the affected regression coefficients are smaller than in Model 1. That is exactly what you would expect.

Also, you may already know that both your models explain very little regarding the behavior of your dependent variable. Model 1 has an R Square of only 0.16; and Model 2 of 0.18. In other words, you derived very little incremental information from adding all those interaction variables within Model 2.

Also, if I look at your variables regression output. The majority of them do not seem statistically significant at any level. And, Model 2 has exacerbated that problem by creating or adding many more non-statistically significant variables.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Can the individuals who have downvoted my answer articulate why they did so? They may have very good reasons for doing so. And, in such case they should share such reasons. $\endgroup$ – Sympa Oct 20 '19 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.