1
$\begingroup$

I'm running/ reporting lmer analysis for the first time and have a hypothetical question related to a few of my models. I'm using R lmer package - my models are generally of the formula:

model=lmer(dependantVar ~ continuousVar*catagoricalVar+  (1|Subject), data=data, REML = FALSE);
anova(model)

In some instances, I have an interaction (sig p value) - this is fairly straightforward as I interpret that and "ignore" the main effects.

However, when I have no interaction, but a main effect of the continousVar only. I'm not sure what to do in terms of reporting the findings. Do I report the main effects estimates from the model above, or do I refit the model without the interaction terms first? i.e.

 modelreduced=lmer(dependantVar ~ continuousVar +  (1|Subject), data=data, REML = FALSE);

or

     modelreduced2=lmer(dependantVar ~ continuousVar + catagoricalVar +  (1|Subject), data=data, REML = FALSE);

and report these estimates.

It's not as if they change a great deal but I'm trying to gage from the literature what people do, and there seems to be examples of both. Although I feel like leaving the interaction in is incorrect, I have come across instances of it being reported.

Any pointers to the "proper way" (if there is one) would be appreciated.

I should say that my main interest is my continuousVar on my dependantVar, but I did believe catagoricalVar may effect the steepness slopes, even by a small amount, so wanted to capture this in the model. So even if it's not significant (if we defer to p values here) should I leave it in anyway?

Also to note, whilst there is a slight reduction in AIC values:

lrtest(model,modelreduced)
lrtest(model,modelreduced2)

suggests no significant difference in these models.

Thank you.

$\endgroup$

1 Answer 1

1
$\begingroup$

Chapter 4 of Frank Harrell's Regression Modeling Strategies covers that and many other issues relevant here. The general strategy is to decide on how many coefficient values you can try to estimate without overfitting ("degrees of freedom"), decide how many degrees of freedom to spend on each predictor (including interactions), then spend those degrees of freedom in the model and report the results.

Thus it's best to decide beforehand on whether to include an interaction, based on your understanding of the subject matter. If you decide to include an interaction, then report the model with the interaction. If you later refit the model based on the "significance" results of the interaction term, then you've used the outcome to choose the (new) model in a way that violates the assumptions behind significance testing.

There is, however, another place where you might want to spend more degrees of freedom: your modeling of continuousVar. As Harrell explains, a strictly linear association of a continuous predictor with outcome is unlikely to hold in practice. A more flexible model, like a regression spline, might be called for; you would include an interaction of the spline with categoricalVar. That would be wise, if your data set is large enough to allow you to spend more degrees of freedom. Again, you would pre-specify the model with the spline and report the results, not re-fitting even if the nonlinearity is "not statistically significant."

$\endgroup$
4
  • $\begingroup$ Thank you @EdM. You have answered my general question... should I stick with my apriori model (yes). I had already been comparing models and excluding some random effects terms to avoid overfitting and perhaps got a bit over obsessed with the concept. Thank you for the recommended reading. I have tried fitting some 2nd order polynomials and it hadn't really improved models - and isn't really an a priori hypothetical assumption ... but I will have at the spline option for reference. Thank you for the quick reply! $\endgroup$
    – JayBee
    Apr 14, 2023 at 16:13
  • 1
    $\begingroup$ @JayBee standard polynomials are seldom helpful, as they try to fit the entire data range at once. Splines provide more localized fits that can better capture internal details. $\endgroup$
    – EdM
    Apr 14, 2023 at 16:15
  • $\begingroup$ Thanks again @EdM, I'll do some more reading. I also have Knoblauch and Maloney's -Modelling Psychophysical Data in R textbook and they recommend R's Spline package for some data sets. It's something I've yet to explore. $\endgroup$
    – JayBee
    Apr 14, 2023 at 16:37
  • 1
    $\begingroup$ @JayBee I would certainly trust recommendations from Professor Maloney. Also consider whether your multiple models are really independent of each other. You might need to consider a multivariate (multiple-outcome) approach if the different outcomes you are modeling are correlated. $\endgroup$
    – EdM
    Apr 14, 2023 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.