Can an inference be made about how an outcome might look with “more power”?

Question

I have a binomial outcome that I have compared across two groups:

Control - 21/783 occurrences (2.68%, 1.71-4.14% 95% CI)

Experimental - 13/261 occurrences (4.98%, 2.79-8.57%, 95% CI)

The p value is greater than 0.05, so it’s “not a significant difference” according to the threshold we established.

Another doctor I work with (MD) said of these particular outcomes:

“Trends are otherwise almost all worse for [experimental] group and [outcome] would almost certainly be significant if numbers were higher.”

He is implying that with more participants - higher power - a “significant difference” definitely would eventually result. Essentially, this is the whole “trend towards significance” thing that papers often like to include in their discussion, but I know rightfully to be fallacious.

How can I rationally explain why this is not necessarily true - that a “trend” towards significance does not necessarily imply significance will be achieved with greater statistical power? Furthermore, is there anyway to quantify the likelihood of obtaining significant results with higher power?

Answer 1

I am a bit surprised that this question did not get more traction ... So, I start the discussion and let's see if it moves further.

It is clear that the small sample size of your experimental group does not help to detect a real effect (if any). To illustrate it, I simulated a bigger experimental sample size but kept the same proportion (i.e. $n_2$ increases but $\hat p_2$ does not change). You did not mention the test statistic that you were using, so I assume that you were performing a "standard" test for two binomial proportions:

$$z=\frac{\hat p_1-\hat p_2}{\sqrt{\hat p(1-\hat p)\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$$ where $\hat p=\frac{n_1\hat p_1+n_2\hat p_2}{n_1+n_2}$.

In this scenario, the hypotheses for this test are:

• $H_0$: the proportions for the two populations are equal ($\hat p_1 = \hat p_2$)
• $H_A$: the proportions for the two populations are not equal ($\hat p_1 \neq \hat p_2$)

So, from the charts below, we can see that as $n_2$ gets bigger, p-value gets lower and power gets bigger (I used bpower from Hmisc package to determine the power):

n1 <- 783; e1 <- 21; p1 <- e1/n1
m <- 1:10
n2 <- 261*m; e2 <- 13*m; p2 <- e2/n2

p <- (p1*n1 + p2*n2)/(n1+n2)
ts = (p1-p2)/sqrt(p*(1-p)*(1/n1 + 1/n2))

pvalue <- pnorm(ts)
power <- bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)
plot(n2, pvalue, type="l", col="red")
plot(n2, power, type="l", col="blue")

For your current study, we have the following power:

n2 <- 261; e2 <- 13; p2 <- e2/n2
bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)

##     Power 
## 0.4495768

So, it has a 45% chance of detecting an effect that exists. But sample size is not the only parameter that impacts the power. Now let's assume that your experimental group has actually double observations with the same proportion (26 successes out of 522 trials):

n2 <- 522; e2 <- 26; p2 <- e2/n2
p <- (p1*n1 + p2*n2)/(n1+n2)
ts = (p1-p2)/sqrt(p*(1-p)*(1/n1 + 1/n2))

pvalue <- pnorm(ts)
pvalue

## [1] 0.01450141

bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)

##    Power 
## 0.583719

By doing this, the power has increased (power = 58%) and therefore we increased our chance to detect an effect that actually exists. We are also in a position where we can reject the null hypothesis.

So, if we assume now that we have the same sample size ($n_2=522$) but instead of having 26 successes we have only 24. The entire test results are different: we are in a position where we cannot reject the null hypothesis and power significantly decreases (power = 46%).

n2 <- 522; e2 <- 24; p2 <- e2/n2
p <- (p1*n1 + p2*n2)/(n1+n2)
ts = (p1-p2)/sqrt(p*(1-p)*(1/n1 + 1/n2))

pnorm(ts)

## [1] 0.03158036

bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)

##     Power 
## 0.4615091

Since power is not only linked to sample size, it would require to maintain the same "success rate" (or more) and a bigger sample (about $n_2 = 1700$ using bsamsize function) to be as affirmative as your colleague. Indeed, we have seen that 2 drops in successes out of 522 (0.4%) is sufficient to change the conclusion of the study.