I have run a mixed effects model with a ln transformed continuous response (seconds) and found a significant effect of the categorical predictor (treatment/control, the only fixed effect in the model).

I want to: 1- Report an effect size (cohen's d, etc) 2- Describe the magnitude of the effect in terms of mean number of seconds that the treated individuals differed from the control individuals, after accounting for the random effects.

I am not sure how to achieve either of these goals. Thank you very much in advance for any advice you can offer.

My code and results are below.

mod1 = lmer(data=data, ln_duration ~ treatment + (1 | id/date/size) +  
(1 | visitor), na.action=na.exclude)

Linear mixed model fit by REML 
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: ln_duration ~ treatment + (1 | id/date/size) +  
(1 | visitor)
 Data: data

REML criterion at convergence: 248

Scaled residuals: 
  Min      1Q  Median      3Q     Max 
-2.7323 -0.4963 -0.0206  0.5600  3.8502 

Random effects:
 Groups                             Name        Variance Std.Dev.
 display_size:(date:id)             (Intercept) 0.00000  0.0000  
 date:id                            (Intercept) 0.00000  0.0000  
 visitor                            (Intercept) 0.03574  0.1891  
 id                                 (Intercept) 0.01164  0.1079  
 Residual                                       0.20001  0.4472  
Number of obs: 170, groups:  
size:(date:id), 130; date:id, 128; visitor, 118; id, 58

Fixed effects:
                      Estimate     Std. Error  df t value  Pr(>|t|)    
(Intercept)               -0.17012    0.06334  41.63000   -2.686 0.010348 *      
       treatmenttreatment  0.31172    0.08135  40.27000    3.832 0.000436 ***
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
    trtmnt -0.729
  • $\begingroup$ I realize that I can probably trim down the random effects (indeed, the results are qualitatively the same and the AICc is lower with only id as a random effect), but I think the answers to my questions above should be the same regardless. $\endgroup$ – JKO Jan 24 '17 at 23:28

Brysbaert and Stevens recently published a paper on how to compute effect sizes with the lme4 package.


library (lme4)
fit <- lmer(RT ~ prime + (prime|item) + (prime|participant), data = data)

There is one fixed effect (the effect of prime) and four random effects:

The intercept per participant (capturing the fact that some participants are faster than others). The intercept per item (capturing the fact that some items are easier than others). The slope per participant (capturing the possibility that the priming effect is not the same for all participants). The slope per item (capturing the possibility that the priming effect is not the same for all items).

I have not been able to use this solution as I used the nmle package because it lets me define the autocorrelation structure (Pinheiro & Bates, 2008) more easily, but I thought I will share it anyway.

| cite | improve this answer | |
  • 1
    $\begingroup$ This answer doesn't show how to use the model to find the effect size, nor does it describe how to interpret that effect size. $\endgroup$ – John Flournoy Jan 31 at 21:13

You can indeed compute an effect size in multilevel models. The one provided is called delta total, where total is the total of the variance components. I generally use it when the co-variate in the model is categorical. It should be close to cohen's d, but I would not call it that. Rather, I would refer to it as an effect size parameter. Computing the interval will be challenging in a frequentist framework, but is easily done using Bayesian methods. Since Bayesian methods provide entire posterior distributions, calculation of delta total is done on the posterior distributions, which readily allows for computing credible intervals via the quantile function in r or some package for obtaining high density intervals.

This is a simple case, however, and I would recommend reading the paper cited for other ways to compute effect sizes in multilevel models.

$$ \delta_t = \frac{beta_{treat}} {sqrt(sigma_{visitor}^2 + sigma_{date:id}^2 +sigma_{display}^2 + sigma_{resid}^2 )} $$

Hedges, L. V. (2007). Effect Sizes in Cluster-Randomized Designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370.

| cite | improve this answer | |
  • $\begingroup$ Thank you for this advice @D_Williams. If I'm understanding the paper you referenced correctly, the delta t effect size is the difference of the group means over the summed standard deviations of the random effects, correct? In that case, I will end up with a single number, and I'm not sure where the interval that you refer to comes in. Any light you could shed on that would be very helpful. As well, Once I have a delta t, how do I interpret it? What is a big delta t, and what is a small one? Again, many thanks! $\endgroup$ – JKO Jan 25 '17 at 19:18
  • 1
    $\begingroup$ @JKO I am referring to computing a confidence interval based on the MLM estimates. The article, I am pretty sure, discusses this at some length. Even if it does not (have not read it in some time), the effect size should have a measure of uncertainty. An effect with interval 0.35 - 0.65 is much different than 0.01 - 1.0, for example (point estimate around 0.5). In Bayesian methods, the interval can be obtained by using the formula provided, but using the posterior estimates. I always state that the interpretation can follow Cohen's d, which are general guidelines that many know of. $\endgroup$ – D_Williams Jan 25 '17 at 19:22
  • $\begingroup$ so it does! I missed that, my apologies. One more question, I am not familiar with the "cluster" terminology. Hedges refers to various sites receiving a single treatment as clusters, and the cluster sample size as the number of experimental units in each site. If all experimental units are located in a single site, then is there effectively just one cluster? As well, would the issue of unequal cluster sample sizes remain a problem, or could the simpler CI equation be used? Thank you again. $\endgroup$ – JKO Jan 25 '17 at 19:49
  • $\begingroup$ @JKO I almost always use Bayesian methods, so can't be much help here. Bayesian has clear advantages in MLM, computation of intervals being one of them. This is not to say obtaining the posterior is trivial, but once obtained everything flows very naturally. See the package brms. $\endgroup$ – D_Williams Jan 25 '17 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.