I am doing simulations of type 1 error, power, and power' (power corrected for anti conservativity) for research on a specific application of (g)lmer, namely to small-N designs of longitudinal repeated measures.
My question is about what it might mean that I am seeing inflated type 1 error (around 0.09) for a main effect of a continuous predictor, but not for its interaction with a categorical variable. To be more specific (and I can provide full details, but I don't think it should be necessary), the model is a binomial (logit) glmer with 6 fixed effects (4 continuous predictors, 1 categorical (coded as -1 or 1) predictor, and an interaction between the categorical and one of the continuos predictors). The RE structure is crossed for items and participants, and it is maximal in the sense that there is correlated random intercept and slopes for all predictors by participant, and for all but 2 of the predictors by item (those 2 predictors are 'nuisance' variables which take on only one level per item).
I am using the simulate function to test type 1 error by using all of the parameters originally fit to the data in the above mentioned model, with the exception of setting two of parameters using newparams= to zero for two of the betas, the main effect of one of the continuous variables and its interaction with the categorical variable.
I am thus running glmer again for each newly simulated data set and storing the p-vals as returned by lmerTest (Satterthwaite... I may also do a LRtest later).
So to get to the point finally, what I have is type 1 error for the interaction is right where I would want to see it, 0.048. But for the main effect it is at 0.09. That's not insanely high, so probably I am going to recommend that in this case we may want to have a lower alpha than 0.05, say 0.01 (in the original model the p-val we observed for this beta is 0.0003 so I'm not really worried about it... and for the interaction it is 0.002). But what might it mean that we see inflated type 1 error for one effect but not another? and in particular when we are talking about a main effect and an interaction? Could this indicate a misspecified RE structure? Should I not worry about this at all? Any responses to this are very welcome, thank you!
