1
$\begingroup$

UPDATE: I don't think the single posted answer is correct. When I run summary(lm(DependentVar ~ IndVar1 * IndVar2, data = Data1)) the first listing is IndVar11 @ p = 0.113, and the four IndVar1:IndVar2 entries are all non significant (p = 0.16 to p = 0.41).

Regarding the suggested duplicate - the accepted answer there is about small p-value differences and small samples sizes, neither of which is the case here. In that answer, the author says he doesn't see other way this can happen. So what is happening here?

I'm looking for an explanation as to how an independent variable can go from being highly significant to being highly insignificant when an interaction term is added. IndVar1, below.

IndVar1 has 2 levels. IndVar2 has 5 levels.

Thanks

> Model_1 <- aov(DependentVar ~ IndVar1 + IndVar2, data = Data1)
> Anova(Model_1, type="III")
Anova Table (Type III tests)

Response: DependentVar
             Sum Sq   Df    F value    Pr(>F)    
(Intercept) 18486.4    1 11622.9984 < 2.2e-16 ***
IndVar1         23.7    1    14.8532 0.0001353 ***
IndVar2    39.7    4     6.3382 5.711e-05 ***
Residuals    2175.1 1341                         
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  
  
> Model_1 <- aov(DependentVar ~ IndVar1 * IndVar2, data = Data1)
> Anova(Model_1, type="III")  
Anova Table (Type III tests)

Response: DependentVar
                    Sum Sq   Df   F value    Pr(>F)    
(Intercept)        15473.4    1 9781.1865 < 2.2e-16 ***
IndVar1                 4.3    1    2.7758 0.1131780    
IndVar2           36.8    4    5.8247 0.0001231 ***
IndVar1:IndVar2    14.1    4    2.2124 0.0666647 .  
Residuals           2149.1 1337                        
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>
Improve this question
$\endgroup$
11
  • $\begingroup$ This is a FAQ: see stats.stackexchange.com/…*%20change. $\endgroup$ – whuber♦ Sep 20 '20 at 17:30
  • $\begingroup$ The accepted answer at "The question already has an answer here" is about small differences in significance, and small samples sizes (using up degrees of freedom). Neither of which is the case here. There is a very large sample, and very large p-value differences. The author of the accepted answer specifically says those are the only cases where he thinks this is possible. So what's happening here? $\endgroup$ – Kyle Sep 20 '20 at 17:39
  • 1
    $\begingroup$ Have you looked at more of the hits in the link? Variations of your question abound on this site, concerning all conceivable permutations of adding or removing variables that increase or decrease significance levels in all kinds of regression models. I'm sure you can find situations essentially the same as yours as well as good advice. One aspect that rarely matters in such circumstances is the amount of data. As far as interactions go, you can find loads of answers pointing out that they usually are almost collinear with their component variables. $\endgroup$ – whuber♦ Sep 20 '20 at 17:44
  • $\begingroup$ No, I didn't. I will go back and read the other anwers. $\endgroup$ – Kyle Sep 20 '20 at 17:46
  • $\begingroup$ I don't think you can center factors, so I don't find any help there. $\endgroup$ – Kyle Sep 20 '20 at 18:44
5
$\begingroup$

In a model with interactions, the interpretation of the IndVar1 term is not 'the effect' of IndVar1, or even 'the association', since the whole point of the interaction is that the association of IndVar1 with the outcome is different for different value of IndVar2.

So, what is it? Well, for Type III sums of squares, it's the estimated association at the value of IndVar2 coded as zero. In this case, if you haven't changed any of the defaults, that will be the estimated association between IndVar1 and the outcome at the reference category (first category) of IndVar2. There's very waek evidence for the association being non-zero

This could happen for multiple reasons, which we can't distinguish because you haven't told us enough

  • maybe the association is actually weak at that level of IndVar2
  • maybe there are very few observations at that level of IndVar2, so you can't tell if the association is actually weak or strong
  • maybe the observations at that level of IndVar2 mostly have the same value of IndVar1, so again you can't tell if the association is actually weak or strong

As an example of the first type, might think of income by age-group and gender, where you could see a strong effect of age, a strong effect of gender, and an interaction -- but if your lowest age group was 0-10, there wouldn't be much gender difference in that age group because none of them are earning much.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Regarding defaults, I ran this to correct Type III sums from car::Anova() : options(contrasts = c("contr.sum", "contr.poly")) $\endgroup$ – Kyle Sep 20 '20 at 9:17
  • $\begingroup$ I don't think this answer is correct. When I run summary(lm(DependentVar ~ IndVar1 * IndVar2, data = Data1)) the first listing is IndVar11 @ 0.113, and the four IndVar1:IndVar2 entries are all non significant (p = 0.16 to p = 0.41). $\endgroup$ – Kyle Sep 20 '20 at 9:29
  • $\begingroup$ This STUPID system won't let me remove my upvote. I'm not too happy about that. $\endgroup$ – Kyle Sep 20 '20 at 17:09
  • 1
    $\begingroup$ @Kyle There are good reasons for that. You will be able to alter your vote if and when this post is edited. Moderators have no power to change or remove votes -- we can't even find out who voted! $\endgroup$ – whuber♦ Sep 20 '20 at 17:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.