I have a multilevel model with 2 levels (L1 = individuals, at least 710 per country; L2 = countries, 17 total)
mod <- lmer(Y ~
pred1 * pred2 +
(1 + pred1 | cluster), data = d)
where
y = continuous outcome variable at L1
pred1 = categorical predictor with levels A and B at L1
pred2 = continuous predictor at L2
cluster = cluster variable with 17 groups.
Now, there is a significant pred1 fixed effect (p< .001), and I was asked to preform a post hoc power analysis arguing that 17 L2 groups do not yield acceptable power.
Questions:
Does it ever make sense to do post hoc power analysis, for either significant or insignificant results? IMO, if the effects of interest are significant, then you are not worried about power (that is, you worry about it if you want to replicate your results). If the effects are insignificant, then estimating observed power is a direct function of the effect's p value (see http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do-if-your.html)
Some recent articles (e.g., Arend & Schafer, 2019) claim that one needs at least 30 clusters at L2 to have any notion of sufficient power for fixed effects (like in the example above), but this is unrealistic for some cases (like, when clusters are big cities in a small country, or even countries). Also, an answer by Robert Long (Minimum sample size per cluster in a random effect model) suggests that the number of clusters is more important for power than the number of units within clusters. I've seen many instances where 10 < n < 20 clusters were modeled within a multilevel model, not as fixed effects. And even if we model them as fixed effects, the problem of power (that is supposedly there) would not magically disappear.
Can you clarify for me why 30 clusters are considered the minimum for sufficient power in multilevel models?
Literature
Arend, M. G., & Schäfer, T. (2019). Statistical power in two-level models: A tutorial based on Monte Carlo simulation. Psychological Methods, 24(1), 1–19. https://doi.org/10.1037/met0000195