1
$\begingroup$

I have a data set with N=18 and a simple linear model y ~ factorX*numericR + (1|id). I now found a significant but small correlation for numericR (-0.24). I did not run a power analysis to determine sample size, as it was not possible to recruit more than 18 participants. I now would like to know if the effect I found is even relevant considering my sample size. So, I had the idea to run a power simulation using the simr package in R:

f1 <- lmer(y ~ factorX*numericR + (1|id), data, control = lmerControl(optimizer ="bobyqa"))

simr::simrOptions(nsim = 100)
efs <- c(seq(from = -0.25, to = 0, by = 0.05))

for(i in 1:length(efs)){

  f1@beta[names(fixef(f1)) %in% "factorX0:numericR"] <- efs[i] 
 
  set.seed(123+i)
  
  (summary(simr::powerSim(f1, test = simr::fixed("factorX0:numericR", "z"))))
}

I am aware that an observed power analysis with the same effect I found is not meaningful. However, I am getting power values of 0.65, 0.53 etc. for smaller effect sizes (-0.20, -0.15). Can I now reason in light of this simulation that my study had at least a power of 0.65 to find an effect of -0.25?

Looking forward to helpful remarks!

Best Pearson

$\endgroup$
2
  • $\begingroup$ Why are you using lmer, not lm? $\endgroup$ Jun 16, 2022 at 20:48
  • 1
    $\begingroup$ Thanks, I forgot to add the varying intercept. $\endgroup$
    – Pearson
    Jun 17, 2022 at 6:01

1 Answer 1

4
$\begingroup$

Your model has (at least) four parameters:

  • intercept
  • main effect for binary(?) predictor X
  • main effect for numeric predictor R
  • interaction between X and R

Yet in your simulation you vary only the size of the interaction effect. The intercept and the main effects are fixed at the values you estimated from your N=18 dataset.

Since the simulation is to a large extent constrained by the data (which generated the hypothesis you are now interested in), this is still an observed power analysis. And it will most likely overestimate the power to detect an interaction effect of -0.25.

It's better to report the coefficient estimates with their standard errors/confidence intervals, which quantify the uncertainty about the true effect sizes.

$\endgroup$
2
  • $\begingroup$ Thank you very much, that was my intuition but I was not sure about it. Is there a way to quantify how much power is overestimated in such an observed power analysis? $\endgroup$
    – Pearson
    Jun 17, 2022 at 10:24
  • 2
    $\begingroup$ You can still do a power simulation, just make it more realistic by simulating datasets of N = 18 under reasonable values for all params. (I think you can do that with simr?) I suspect that you'll get strong indication that the power is much lower than 60%. Computing the power of an experiment that already happened seems a bit pointless. It might be more productive to conclude what you've learned from this experiment and think about a followup experiment. $\endgroup$
    – dipetkov
    Jun 17, 2022 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.