2
$\begingroup$

I'm trying to find the minimum detectable effect size (MDES) given my sample, alpha (.05), and desired power (90%) in a linear mixed model setting. I'm using the simr package for a simulation-based approach. What I did is changing the original effect size to a series of hypothetical effect sizes and find the minimum effect size that has a 90% chance of producing a significant result. Below is a toy code:

library(lmerTest)
library(simr)

# fit the model
model <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)
summary(model)

Model's Fixed effect:

Fixed effects:
            Estimate Std. Error      df t value Pr(>|t|)    
(Intercept)  251.405      6.825  17.000  36.838  < 2e-16 ***
Days          10.467      1.546  17.000   6.771 3.26e-06 ***

Here is the code for minimum detectable effect size:

pwr <- NA

# define a set of reasonable effect sizes
es <- seq(0, 10, 2)


# loop through the effect sizes
for (i in 1:length(es)) {
  # replace the original effect size with new one
  fixef(model)['Days'] =  es[i]
  # run simulation to obtain power estimate
  pwr.summary <- summary(powerSim(
    model,
    test = fixed('Days', "t"),
    nsim = 100,
    progress = T
  ))
  # store output
  pwr[i] <- as.numeric(pwr.summary)[3]
}


# display results
cbind("Unstandardized Regression Coefficient" = es,
      Power = pwr)

Output:

     Unstandardized Regression Coefficient Power
[1,]                                     0  0.09
[2,]                                     2  0.24
[3,]                                     4  0.60
[4,]                                     6  0.99
[5,]                                     8  1.00
[6,]                                    10  1.00

My questions:

(1) Is this the right way to find the MDES?

(2) I have some trouble making sense of the output. Can I say the following: because the estimated power when the effect = 6 is 99%, and because the actual model has an estimate of 10.47, then the study is sufficiently powered? Conversely, imagine that if the actual estimate was 3.0, then can I say the study is insufficiently powered?

$\endgroup$
5
  • $\begingroup$ I think you'll want to specify a different type of approximation, it's been about a year since I've used simr but by default it didn't used to use the Kenward-Roger or Satterthwaite approximations, which have been shown to be preferable: Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior research methods, 49(4), 1494-1502. $\endgroup$
    – sjp
    Commented Jul 4, 2020 at 1:20
  • 1
    $\begingroup$ I'm a bit confused. Have you already done the study and collected data and now you are doing a power analysis? $\endgroup$ Commented Jul 4, 2020 at 5:18
  • $\begingroup$ @RobertLong Unfortunately, yes. It was required by a reviewer. Is there any scenario this analysis can be justified (if the code is correct)? For example, if this was a pilot study and I want to do an exact replication (same N), can it inform me whether effects in my replication have suffiicent power or not? $\endgroup$
    – Han Zhang
    Commented Jul 4, 2020 at 14:43
  • 1
    $\begingroup$ How did you determine the size of data that you collected? This is a controversial topic. One common view is that you should ignore the effect size that you found. Andrew Gelman put’s it well: "It’s fine to estimate power (or, more generally, statistical properties of estimates) after the data have come in—but only only only only only if you do this based on a scientifically grounded assumed effect size. One should not not not not not estimate the power (or other statistical properties) of a study based on the 'effect size observed in that study. That’s just terrible" $\endgroup$ Commented Jul 4, 2020 at 15:42
  • 1
    $\begingroup$ @RobertLong I didn't do a power analysis a priori and just tried to collect more N than what is typically done in the field. I understand posthoc power is useless. But I also recall that one can do a sensitivity analysis after data comes in, to determine the minimum effect that would be reliably (90%) significant. $\endgroup$
    – Han Zhang
    Commented Jul 4, 2020 at 16:00

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.