1
$\begingroup$

I would like to know how to do a sample size calculation without data/simulation for mixed-effect models.

To briefly explain my research, I want to investigate which method (A or B) is more effective for promoting students' participation in which the students are grouped for discussion.

  • 2 between-subject condition: A or B
  • 1 dependent variable: participation
  • 1 random-effect variable: group

I will build two mixed-effect linear models and conduct 'ANOVA' to get the significance of the condition:

  • modelA = participation ~ condition + (1|group)
  • modelB = participation ~ 1 + (1|group)
  • anova(modelA, modelB)

Q1. Would it be fine to conduct sample size calculation using simple ANOVA-based power analysis (like G*Power)? In this case, the random-effect won't be considered, and was wondering if this is important. If it is important, how much difference in sample size would it make? If it is not important, why?

Q2. I know that there are some programs like simr that is for mixed-effect models, but they require pilot data or estimations for data parameters. I don't have those data and I only have 3 parameters with conventional values: d=0.5, a=0.05, power=0.8. Are there any method that I can get a sample size using these three parameters?

Thank you in advance!

$\endgroup$

1 Answer 1

1
$\begingroup$

Q1. Would it be fine to conduct sample size calculation using simple ANOVA-based power analysis (like G*Power)? In this case, the random-effect won't be considered, and was wondering if this is important. If it is important, how much difference in sample size would it make? If it is not important, why?

No, it is not fine to ignore the random effects. The greater the intra-class correlation, the lower will be the effective sample size.

Q2. I know that there are some programs like simr that is for mixed-effect models, but they require pilot data or estimations for data parameters. I don't have those data and I only have 3 parameters with conventional values: d=0.5, a=0.05, power=0.8. Are there any method that I can get a sample size using these three parameters?

Well, you could do the simulations from scratch by yourself, but you are still going to need values for all relevant parameters. If you don't have them then you can use a range of estimates from other studies, or from expert domain knowledge.

$\endgroup$
4
  • $\begingroup$ Thank you for your reply! I have a follow-up question regarding what you said " The greater the intra-class correlation, the lower will be the effective sample size.". Then, if I calculate the sample size without considering the random effect and use that sample size for the study, wouldn't it mean that I have a sufficient sample size? Because adding random effect would only decrease the sample size. $\endgroup$
    – gina
    Jan 11, 2021 at 23:09
  • $\begingroup$ You're welcome. No, that's not correct. Suppose that we conduct a power analysis for a model without random effects and determine that the sample size needed to detect the effect size you are interested in is 1000. Now, if we introduce random effects, the effective sample size might now by only 700, so we would have insufficient power becuase our effective sample size is below 1000. $\endgroup$ Jan 12, 2021 at 10:56
  • $\begingroup$ Sorry, I am a bit confused. In your example, 1000 is larger than 700 so conducting a study with 1000 samples will give me sufficient power? Because adding more samples will increase power. I am guessing the only thing that I should worry about having more samples than 700 is overpowered results? $\endgroup$
    – gina
    Jan 14, 2021 at 0:31
  • $\begingroup$ No. If you have 1000 actual samples, but the effective effective sample size is only 700 then you would need to increase the sample size so that the effective sample size is 1000. $\endgroup$ Jan 14, 2021 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.