I have two "unrelated" data frames (df1, and df2). The type and number of columns and rows is identical, just the actual data differs.
I am fitting the same base and full models over both:
df1_base_model = lmer(Time~1 + (1|Question) + (1|Participant), df1, REML=FALSE) df1_full_model = lmer(Time~Condition + (1|Question) + (1|Participant), df1, REML=FALSE) df2_base_model = lmer(Time~1 + (1|Question) + (1|Participant), df2, REML=FALSE) df2_full_model = lmer(Time~Condition + (1|Question) + (1|Participant), df2, REML=FALSE)
To see whether the introduction of
Condition as a fixed factor makes any difference, I run:
anova(df1_base_model, df1_full_model) anova(df2_base_model, df2_full_model)
and get this output for df1:
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq) df1_base_model 4 2028.2 2045.0 -1010.1 2020.2 df1_full_model 6 2024.6 2049.9 -1006.3 2012.6 7.5162 2 0.02333 *
and for df2:
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq) df2_base_model 4 1402.0 1418.9 -697.02 1394.0 df2_full_model 6 1280.9 1306.1 -634.45 1268.9 125.14 2 < 2.2e-16 ***
According to the output, the introduction of
Condition is statistically significant for both df1 and df2. But I need some way to quantify the effect, beyond just a p value. The data is such that the introduction of
Condition has only a marginal effect in df1 but a very large effect in df2, and the difference of p values already suggest that, but I need a better effect size measure.
So my question is: What is the proper way to compute and compare effect sizes in this situation? I've read that many traditional measures, such as Cohen's d or (partial/generalized) eta squared do not apply to linear mixed models with random effects. So what ways are there to compare the models for df1 and df2 and show that the effect size of introducing
Condition is much larger for df2?
EDIT: In case it helps, my output variable
Time is measured in seconds. Perhaps there is a way to express the effect size in terms of seconds somehow?