2
$\begingroup$

I have a study in which ~60 participants rating a subset of 200 items. I have four potential predictors that I would like to use to predict those ratings.

I will run an lmer() model including random subject and item intercepts.

Is there any harm in simply including all four candidate predictors, and seeing which are significant using the "lmerTest" package?

I had originally thought to build the model one predictor at a time via LRTs, starting with the one I have the most reason to believe should matter. But I was told this is a bad approach.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Is there any harm in simply including all four candidate predictors, and seeing which are significant using the "lmerTest" package?

Breaking that into two questions, the answers are respectively "no" and "yes."

Separating out the different predictors throws away shared information and can lead to problems like omitted-variable bias. Sometimes the effect on one predictor might depend directly on the value of another, or the influence of one predictor might best be seen when other predictors are also taken into account. The only harm in "including all predictors" is when you have a low ratio of cases to predictors, which is not your problem here. Even then, there are principled ways to proceed that effectively penalize predictors to avoid the overfitting that would otherwise ensue. So there's no harm in including all 4 of your predictors at once.

You should, however, reconsider the part about "seeing which are significant using the 'lmerTest' package." If your interest really is in prediction and you can expect to have values for all 4 of your predictors in future cases, there's no reason to cut back on the number of predictors. That will just tend to make predictions less reliable. In general, for prediction you want to include as many predictors in your model as is consistent with not overfitting. Finding a predictor "not significant" might just mean that your study was too small to document its effect, and if your model is used at a larger scale it might be seen as quite important. Statistical significance is not to be confused with practical importance, in either direction.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks very much for the answer! I'm not so much interested in future cases, but rather to know if/which of the four predictors are related to my dependent variable. Does that change anything? So, I'm not saying I'll remove non-significant predictors, but rather will just not interpret them. $\endgroup$
    – Dave
    Aug 7, 2020 at 22:49
  • $\begingroup$ @Dave sill include all predictors in the model, at least to start. Do temper any statements you might make about the practical importance of the "not-significant" predictors. It might just be that your study wasn't big enough to find their significance. Also, make sure that omitting the "not-significant" predictors doesn't substantially affect conclusions about the others. It's possible that removing a "not-significant" predictor will affect your ability to evaluate the others adequately. I'd just report the entire model, and let the significance estimates speak for themselves. $\endgroup$
    – EdM
    Aug 7, 2020 at 22:55
  • $\begingroup$ Thanks a lot @EdM ! $\endgroup$
    – Dave
    Aug 7, 2020 at 23:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.