3
$\begingroup$

I have a multilevel model with 2 levels (L1 = individuals, at least 710 per country; L2 = countries, 17 total)

mod <- lmer(Y ~ 
               pred1 * pred2 +  
               (1 + pred1 | cluster), data = d)

where
y = continuous outcome variable at L1
pred1 = categorical predictor with levels A and B at L1
pred2 = continuous predictor at L2
cluster = cluster variable with 17 groups.

Now, there is a significant pred1 fixed effect (p< .001), and I was asked to preform a post hoc power analysis arguing that 17 L2 groups do not yield acceptable power.

Questions:

  1. Does it ever make sense to do post hoc power analysis, for either significant or insignificant results? IMO, if the effects of interest are significant, then you are not worried about power (that is, you worry about it if you want to replicate your results). If the effects are insignificant, then estimating observed power is a direct function of the effect's p value (see http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do-if-your.html)

  2. Some recent articles (e.g., Arend & Schafer, 2019) claim that one needs at least 30 clusters at L2 to have any notion of sufficient power for fixed effects (like in the example above), but this is unrealistic for some cases (like, when clusters are big cities in a small country, or even countries). Also, an answer by Robert Long (Minimum sample size per cluster in a random effect model) suggests that the number of clusters is more important for power than the number of units within clusters. I've seen many instances where 10 < n < 20 clusters were modeled within a multilevel model, not as fixed effects. And even if we model them as fixed effects, the problem of power (that is supposedly there) would not magically disappear.
    Can you clarify for me why 30 clusters are considered the minimum for sufficient power in multilevel models?

Literature

Arend, M. G., & Schäfer, T. (2019). Statistical power in two-level models: A tutorial based on Monte Carlo simulation. Psychological Methods, 24(1), 1–19. https://doi.org/10.1037/met0000195

$\endgroup$
5
  • 3
    $\begingroup$ Your point 1 is addressed in many posts here, for example here, although there can be some subtleties as in this question. For point 2, could you please provide references and, ideally, links to those articles? It's hard to clarify an argument that isn't available to examine in detail. $\endgroup$
    – EdM
    Apr 28, 2022 at 13:16
  • 1
    $\begingroup$ For point 1, I heartily recommend the Hoenig & Heisey paper referred to at the question Edm links to. $\endgroup$ Apr 28, 2022 at 13:18
  • $\begingroup$ Thanks for the links, I was wondering whether I was missing something in the multilevel context, but I am pretty confident that the answer is - no. Yep, Hoenig & Heisey paper is good! Re point 2=> I added the literature $\endgroup$
    – User33268
    Apr 28, 2022 at 19:25
  • $\begingroup$ How many individuals are included in your 17 countries? $\endgroup$
    – EdM
    Apr 28, 2022 at 21:35
  • $\begingroup$ All clusters had more than 710 individuals in them. $\endgroup$
    – User33268
    Apr 29, 2022 at 7:16

1 Answer 1

2
$\begingroup$

Answering in reverse:

Question 2. First, the examples in the cited paper on power estimates are for large numbers of clusters, each of relatively small size. The plots of minimum detectable effect sizes (MDES) (Figures 1,2,3) only go up to 30 individuals per cluster and 150 clusters: at most 4500 individuals. You have a different situation: 17 clusters with at least 710 individuals each, over 12,000 individuals. That's more cases than any scenario I saw in a brief look at the paper. The total number of cases typically matters most.

Second, your model seems much less complex than the one illustrated in that paper. The model in the paper allows for a large number of potentially correlated random effects. For example, consider what would happen if your model included random effects for both your fixed-effect terms and for their interaction. The last model in the Mike Lawrence answer on the lmer cheat sheet is for such a model: it requires 14 coefficient estimates. In your case, the constant value of pred2 within each L2 level allows you to omit that as a random effect among clusters. You also omitted the pred1:pred2 interaction as a random effect,* further simplifying the model from what it might have been. I think your model only needs to estimate 6 coefficients.

Third, with only 17 clusters you will nevertheless have imprecise estimates of the variance and covariance of the random intercepts and slopes among clusters.

Question 1. Some recommendations to do "post-hoc power analysis" might not be so useless as they seem at first glance. An a priori power estimate to design a study for mixed-model analysis necessarily makes a lot of assumptions, as the linked paper explains. Re-examining those assumptions based on the observed data can provide both resolution of prior misconceptions and guidance for designing future studies. One might quibble over the terminology of "post-hoc power analysis." Whatever you call it, however, it's a good idea to evaluate what went right and what went wrong after you complete a study. For complicated mixed models, there is probably no better choice for that than simulations, as performed for example with the simr package used in the paper.


*I'm not sure about the wisdom of this omission, but I confess to having a lot of problems thinking about mixed models with interactions.

$\endgroup$
2
  • $\begingroup$ The problem is that I keep getting info that it's not the total number of cases that matters the most, but actually the number of clusters - I added the answer by Robert Long regarding that issue. My main concern is whether the number of clusters influences the power of estimating fixed effect and whether there are some references that deal with that question. Btw. you are right about the omitted interaction! $\endgroup$
    – User33268
    May 2, 2022 at 9:11
  • $\begingroup$ @User33268 yes, but see his answer here to put the number-of-clusters issue into perspective: "some consensus around 6 being perhaps the minimum." 17 clusters should be OK. You can take his advice in that answer and compare 2 models treating the clusters as fixed versus random effects: "If there is meaningful differences then you would have quite an interesting problem. In that case I would probably report the results of both models." $\endgroup$
    – EdM
    May 2, 2022 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.