4
$\begingroup$

I have an experiment with 8 binary IV, 1 three-level IV (all categorical) and a continuous DV. I run an lmer with all main effects of all variables like so:

main_effects<-lmer(agreement~ dir+coref+fuzzy+B_atom+A_atom+A_neg+B_neg+A_qua+B_qua+(1|Index), data=data)

By displaying the summary of that model, I get the following, where factors dir, coref, B_atom, A_neg and B_neg are significant:

Type III Analysis of Variance Table with Satterthwaite's method
        Sum Sq Mean Sq NumDF  DenDF F value    Pr(>F)    
dir    13764.8  6882.4     2 282.50  8.5386 0.0002509 ***
coref   6487.7  6487.7     1 190.33  8.0488 0.0050465 ** 
fuzzy   2538.6  2538.6     1 190.52  3.1495 0.0775475 .  
B_atom  4502.8  4502.8     1 357.59  5.5863 0.0186359 *  
A_atom  2234.7  2234.7     1 357.66  2.7724 0.0967781 .  
A_neg   8802.7  8802.7     1 366.43 10.9209 0.0010446 ** 
B_neg   8995.8  8995.8     1 366.44 11.1605 0.0009215 ***
A_qua     17.3    17.3     1 381.60  0.0215 0.8835529    
B_qua     36.5    36.5     1 381.60  0.0453 0.8315259

However, if I try to run another lmer containing interactions like so

interaction<-lmer(agreement~ dir*coref*fuzzy*B_atom*A_atom*A_neg*B_neg*A_qua*B_qua+(1|Index), data=data)

the main effects showing in the first model completely disappear and I only get some few interactions (I only include the siginifcant interactions in the below output to keep this post as short as possible):

fuzzy:B_atom                              8605.7  8605.7     1 309.35 11.3420 0.0008535 ***    
dir:A_neg                                 5929.7  2964.8     2 316.77  3.9075 0.0210654 *     
dir:B_neg                                 6676.6  3338.3     2 317.00  4.3998 0.0130391 *      
dir:coref:fuzzy                           6911.4  3455.7     2 307.90  4.5545 0.0112381 *

Why do the main effects disappear? If there are interactions of those factors on top of the simple main effects of model 1, shouldn't the main effects be included in model 2 as well? I would understand it if it would have been the other way around: no simple main effects on model 1 but main effects and interactions on model 2. But here I have the opposite situation, which I don't quite get. Most importantly, do I go with the first model or the second one or in other words, would my results be unreliable if I decide to report on the results of model 1 (with main effects only)?

I don't have much experience in this kind of modeling so I am a bit lost right now.

UPDATE I checked for multicollinearity and it seems that this is not a problem:

        GVIF Df GVIF^(1/(2*Df))
dir    1.217513  2        1.050433
coref  1.458521  1        1.207692
fuzzy  1.496577  1        1.223347
B_atom 1.099159  1        1.048408
A_atom 1.098381  1        1.048037
A_neg  1.702295  1        1.304720
B_neg  1.702118  1        1.304652
A_qua  1.534520  1        1.238758
B_qua  1.534869  1        1.238898

From the reading I did, I found that the GVIF can be used with the same rule of thumb as the VIF: everything below 5 is not highly correlated. Should I then stick to my first model with the main effects as the "reliable" one without correlations? Or any other suggestions?

Improve this question
$\endgroup$
1
  • $\begingroup$ What happens to the signs of the "main effects" after adding the interactions? Multicollinearity is a full picture rather than a single number to check (a low VIF doesn't rule out multicollinearity). $\endgroup$
    – LSC
    Apr 4, 2019 at 20:05

3 Answers 3

Reset to default
1
$\begingroup$

Putting aside the issues with p-value calculations in mixed models (see this page and its links), there are a couple of things that could be going on, one of which seems more likely in your case.

First, models with interactions can use up a lot of degrees of freedom. In addition to the 10 degrees of freedom for the specified predictors in your first model (plus 1 for the intercept or grand mean and more for the random effect), each of their two-way, 3-way,...,9-way interactions is an additional parameter value that the second model needs to evaluate, diminishing the residual degrees of freedom. The resulting higher standard errors in coefficient estimates in a model with interactions thus can move a coefficient from "significant" to "non-significant" even if the point estimate of the coefficient is the same.

That doesn't seem to explain your result here, though. The estimated denominator degrees of freedom were typically in the high 300's in the first model, but still in the low 300s for the second model.

More likely here is the second possibility: that the interactions themselves are the interesting results. Look at the predictors involved in the interaction terms you chose to display: dir, core, fuzzy, B_atom, A_neg, B_neg. Those are all of the predictors that were significant, plus the one that came closest to the significance cutoff, in the first model. What the first model told you was that each of those predictors was related to outcome, if you hold constant (ignore) the values of all the other predictors. What the significant interaction terms in the second model told you was that you cannot fairly ignore the values of other predictors when you consider the relation of those individual predictors to outcome.

In that context of important interactions, the "significance" of any "main effect" doesn't really matter: it's the particular combinations of values of dir, core, fuzzy, B_atom, A_neg, and B_neg that matter. Those "main effects" didn't "disappear"; they just didn't reach a threshold of statistical significance on their own. You still need to know the values of the "main-effect" estimates to predict the outcome at any specified combination of the predictors.

Complicating the issue is that different software can report p-values for categorical predictors differently when there are interactions. Sometimes they are reported for a situation where all other predictors are at their reference levels. In that case it's quite possible to get a "non-significant" p-value for a predictor in that scenario, even if is highly associated with outcome at a non-reference level of another predictor due to an interaction. Other software includes a report of p-values for a predictor that includes all of its interactions. So you have to understand how the particular software you are using is presenting its p-values before you can even make the (often arbitrary) decision that the predictor is "non-significant."

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your detailed answer. "In that context of important interactions, the "significance" of any "main effect" doesn't really matter: it's the particular combinations of values of dir, core, fuzzy, B_atom, A_neg, and B_neg that matter. ": so can I say that these specific variables influence my outcome but they also influence the outcome in specific combinations between them? For your other point, I am using R for those calculations and I am not really familiar with other software. $\endgroup$
    – katerinaka
    Apr 5, 2019 at 14:54
  • $\begingroup$ @katerinaka even different R packages can report the p-values in different ways. So you need to know how the particular R package/function you are using makes its reports. The best way to present your results is that those variables influence the outcome when present in particular combinations that your significant interaction terms identified. $\endgroup$
    – EdM
    Apr 5, 2019 at 16:39
  • $\begingroup$ Thanks! This make sense. I will investigate how my particular R package (lme4 and lmerTest) behave and try to reformulate my model and hypotheses in a clearer way to be sure I can interpret the results in a meaningful way. $\endgroup$
    – katerinaka
    Apr 8, 2019 at 7:33
1
$\begingroup$

I suspect that you have a high degree of multicollinearity present in the model, meaning that some (or all) of your independent variables are highly correlated, especially when you add the interaction terms (by construction, interaction terms will be correlated with main effect variables). In the presence of multicollinearity coefficient estimates can change drastically based on which other variables are included in the model (i.e., coefficients become very sensitive to small changes in the model). And multicollinearity reduces the precision of coefficient estimates, which means you might not be able to trust the p-values to determine which variables are statistically significant. Given that a lot of your estimates change, it likely means that most of your independent variables are correlated.

Your next steps will depend on what you are trying to get out of your model and which estimates you need to use. But I recommend checking for multicollinearity by calculating a Variance Inflation Factor (VIF) for each independent variable. I would do this for your first model first and go from there.

There are other posts and articles that go more in-depth on this topic, including:

https://statisticsbyjim.com/regression/multicollinearity-in-regression-analysis/

How to test and avoid multicollinearity in mixed linear model?

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot for your feedback! I could indeed say that some of my independent variables might be highly correlated so I will need to look into that with the VIF you recommended and see. Thanks also for the pointers. I will check these things and get back to you with more comments. $\endgroup$
    – katerinaka
    Apr 4, 2019 at 6:13
0
$\begingroup$

As a general idea, if you test for interactions, and the interaction test is significant, it isn't logical to test the "main effects" or any lower order terms nested in those interactions. By definition and conclusion of your interaction test, you said the variables are statistically useful for predicting Y, so it doesn't make sense to then test the main effects because you already have your answer.

For example, a model of

Y~ A + B + C + D + AB + AC + BC + ABC + error

If interaction of ABC was significant, it doesn't make sense to then go test AB, AC, BC, A, B, or C, but you could test D since D isn't "nested" within the ABC interaction. By definition of your test, you concluded that the relationship of Y with A depends on the combination of B and C (B*C product), for example, so testing the lower order terms is illogical, generally speaking; failure to follow this idea can lead to uninterpretable or seemingly contradictory results.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the pointer! The thing is that I started by first testing the simple model with only main effects and then moved up to the more complex with interaction. And because I had seen the significance of the variables in the simpler model, I started wondering why it disappeared in the more complex one. If I had tested the more complex one first, I would indeed never have seen the difference of the results. $\endgroup$
    – katerinaka
    Apr 5, 2019 at 14:56
  • $\begingroup$ Still with that information, the point remains: you're looking at the pvalue for, say A, and asking "after I account for ABC, AB, AC, BC, B, and C, is A statistically useful in predicting Y?" Think about the question for a minute-- it's a nonsensical question because you're assuming A is useful for predicting Y by accounting for the other terms that already involve A, so if you conduct this question, you can end up with nonsensical "results", as you have. VIF's are also not the be-all-end-all in assessing multicollinearity, so it's totally possible that's playing a role. 1/2 $\endgroup$
    – LSC
    Apr 5, 2019 at 15:18
  • $\begingroup$ 2/2 Perhaps posting the full output (coefficients, coefficient standard errors, CI limits and p-values) for each the reduced and the complete model so we can see more detail of what you mean by "completely disappear". The point of my post is you're asking why you get a weird answer when asking a question that doesn't make practical sense to ask. It's okay that you tried a simpler model first and then added interactions, but it doesn't make sense to look at the nested main effects individually as you have done after you added interaction and found the interaction to be significant. $\endgroup$
    – LSC
    Apr 5, 2019 at 15:21
  • $\begingroup$ Thanks! I will try to rebuild my whole model by thinking about the hypotheses/interactions expectations harder. I will then pose the entire output if required. $\endgroup$
    – katerinaka
    Apr 8, 2019 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.