Why are effect size estimates (such as $R^2$) difficult in mixed-effects models?

I have been reading a lot online about estimating standardised effect sizes in mixed effects models and it seems like there are formidable challenges even for something relatively broad like an $$R^2$$, let alone for contrasts. What is a simple explanation (that I could supply to a reviewer) for why effects sizes are so challenging to perform in these more complex models?

1 Answer

There is nothing inherently challenging. What is challenging is getting a consensus on how many degrees of freedom a LMEM has and what an $$R^2$$ is supposed to reflect in a LMEM. As soon as two parties agree on these matters, the coefficients of determination and effect sizes are "trivial" to calculate. That's why we have all those metrics like $$\Omega^2$$ (Xu, 2003), pseudo-$$R^2$$ (Hössjer, 2008, or Nakagawa & Schielzeth, 2013) and approximations like Satterthwaite's (Satterthwaite, 1946), Kenward-Roger's (Kenward & Roger 1997), etc. The Journal of Statistical Software article on lmerTest offers an excellent discussion on the matter. To that extent, Douglas Bates, the linear mixed effects models' OG, has expressed some of his thoughts on the matter of R2 measure in mixed models online. I fully agree.

• Thank you @user11852. What about standardised effect sizes for custom contrasts within mixed-effects models? They seem to be much harder to obtain than for single-level models? – llewmills Oct 2 '18 at 0:17
• Also, loved 'OG'. – llewmills Oct 2 '18 at 0:38
• Cool! Regarding your comment's question: In general, custom (or Helmert) contrasts are context-specific analysis constructs and we need to carefully assess how many degrees of freedom they dictate. Nevertheless, the custom nature of the contrast is not the main issue; as before it is the inclusion of the random effects that complicates our problem's definition. – usεr11852 Oct 2 '18 at 18:17
• Yes I figured the random effects were the problem. I am mounting an argument to the reviewer than the unstandardised regression coefficients are a better indicator of the size (rather than the noticeability) of the effects in question than a standardised effect size. Thanks for the references! – llewmills Oct 2 '18 at 20:40