Power Analysis result weird for multiple regression in R

Question

I have a glmer model:

glmer(kept ~ agent_liking*.loneliness + (1|participant_id),family = "binomial", data = d)

We collected 150 participants for study 1 and the model was significant. Here's the result of the model:

 Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod']
 Family: binomial  ( logit )
Formula: kept ~ agent_liking * .loneliness + (1 | participant_id)
   Data: d
     AIC      BIC   logLik deviance df.resid
 36433.8  36474.6 -18211.9  36423.8    26275
Scaled residuals:
     Min       1Q   Median       3Q      Max
-1.08217 -1.00014  0.00108  0.99979  1.08470
Random effects:
 Groups         Name        Variance  Std.Dev.
 participant_id (Intercept) 4.118e-17 6.417e-09
Number of obs: 26280, groups:  participant_id, 146
Fixed effects:
                           Estimate Std. Error z value Pr(>|z|)
(Intercept)               0.0654555  0.0646308   1.013   0.3112
agent_liking             -0.0003057  0.0002687  -1.138   0.2552
.loneliness              -0.0644539  0.0348978  -1.847   0.0648 .
agent_liking:.loneliness  0.0003024  0.0001447   2.090   0.0366 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ‘ ’ 1
Correlation of Fixed Effects:
            (Intr) agnt_l .lnlns
agent_likng -0.878
.loneliness -0.917  0.804
agnt_lkng:.  0.806 -0.916 -0.879
optimizer (Nelder_Mead) convergence code: 0 (OK)
boundary (singular) fit: see ?isSingular

As we are using a significant model to conduct a power analysis to figure out the sample size we need for study 2, we expected to see a sample size that is similar to that of study 1 (which is 150).

However, the power analysis showed that we need about 300 samples to reach 80% power.

PowerModel = extend(PowerModel, along = "participant_id", n = 300)
powerCurve(PowerModel,test = fixed("agent_liking:.loneliness"),nsim = 200, along = "participant_id")

    Power for predictor 'agent_liking:.loneliness', (95% confidence interval),
by number of levels in participant_id:
      3:  3.50% ( 1.42,  7.08) - 540 rows
     36: 13.50% ( 9.09, 19.03) - 6480 rows
     69: 25.00% (19.16, 31.60) - 12420 rows
    102: 40.00% (33.15, 47.15) - 18360 rows
    135: 47.50% (40.41, 54.66) - 24300 rows
    168: 62.00% (54.89, 68.75) - 30240 rows
    201: 72.00% (65.23, 78.10) - 36180 rows
    234: 77.00% (70.54, 82.64) - 42120 rows
    267: 80.50% (74.32, 85.75) - 48060 rows
    300: 83.00% (77.06, 87.93) - 54000 rows

I have never had a situation in which a significant effect could not lead to any power. Has anyone experienced this before? Any ideas on what might be the reason? Thanks!

Answer 1

If your p-value is 0.05, then your power (for the same sample size) is 50%.

So this looks about right to me. Your p-value (0.0366) is a little lower than 0.05, so you should get slightly more than 50% power for the same sample size.

Approximate rule of thumb: p-value of 0.01 gives power of about 80%.

Here's code with a similar example, using a t-test:

d <- 0.34

y1 <- scale(rnorm(75))
y2 <- scale(rnorm(75)) + d

t.test(y1, y2)

power.t.test(n = 75, delta = d, sd = 1)
power.t.test(power = 0.8, delta = d, sd = 1)

A p-value of 0.39 gives power of 0.54, and a sample size of 274 (137 per group) is required for 80% power.