Testing the significance of a main effect using model comparison, when an interaction is included

I am fitting a linear mixed effect model (using lmer), and want to obtain p-values via model comparison. As far as I understand it, the procedure is: (i) create a nested model identical to the big model minus one of the variables you want to test; (ii) compare the big model to the nested model, the p-value of this comparison is attributed to the excluded variable; (iii) repeat this for each of the variables (i.e. each nested model is compared to the big model).

What happens when I test for a main effect, and the model includes an interaction term of this factor with some other factor? I learned from other threads it is wrong to have higher-level terms without lower-level ones (this one, or this one, but my mathematical understanding is limited). And yet, it seems that model comparison is standardly done to obtain p-values of main effects even when interaction terms are included (e.g. by using the function mixed() from the afex package with the LRT method, or manually constructing a nested model with an itneraction but without a main effect).

My question is actually twofold:

1. Is it wrong to rely on model comparison for p-values of main effects when I have an interaction? Or does the function mixed() (with the LRT method) do something other than the procedure described above?

2. Assuming all contrasts are orthogonal, I don't understand why it is wrong to have an interaction without a main effect. With orthogonal contrasts, isn't the variability explained by interaction independent of the variability of the main effects?

• When the interaction between A and B exists, it is hard to define the "main effects" of A and B. So there is nothing to estimate/test. Of course, if you can define them, you can estimate/test them. – user158565 Nov 25 '18 at 16:38
• The main effects are defined as the mean effect across all the levels of the other effect. But how does this answer my question regarding a nested model which is identical to the big model in everything but a main effect? – Galit Nov 25 '18 at 17:48
• Suppose 2 factors X and Y, X has level a and b, Y has level 1 and 2 and interaction exist. For response variable Z, the means for 4 different combination of X and Y are $\mu_{a1}, \mu_{a2}, \mu_{b1}, \mu_{b2}$. How to define the main effect of X and Y? – user158565 Nov 25 '18 at 18:01
• It depends on the coding, but assuming sum coding (as I did), and a balanced sample, the main effect of X is the contrast between (μa1+μa2)/2, or -(μb1+μb2)/2. Likewise, for Y: (μa1+μb1)/2, or -(μa2+μb2)/2. – Galit Nov 25 '18 at 18:48
• if $\mu_{a1} = -1, \mu_{a2} =1, \mu_{b1} = 1, \mu_{b2} = -1$, then $(\mu_{a1}+\mu_{a2})/2 = 0$ and $-(\mu_{b1}+\mu_{b2})/2 =0$. Is it you want? Mathematically, it is correct; in practice, it always misleads. – user158565 Nov 25 '18 at 19:04

I don't know what the function mixed does, but if you have an interaction then the test of the main effect is the effect when other variable(s) in the interaction are 0. It is not the usual main effect.