I have a binomial outcome that I have compared across two groups:

Control - 21/783 occurrences (2.68%, 1.71-4.14% 95% CI)

Experimental - 13/261 occurrences (4.98%, 2.79-8.57%, 95% CI)

The p value is greater than 0.05, so it’s “not a significant difference” according to the threshold we established.

Another doctor I work with (MD) said of these particular outcomes:

“Trends are otherwise almost all worse for [experimental] group and [outcome] would almost certainly be significant if numbers were higher.”

He is implying that with more participants - higher power - a “significant difference” definitely would eventually result. Essentially, this is the whole “trend towards significance” thing that papers often like to include in their discussion, but I know rightfully to be fallacious.

How can I rationally explain why this is not necessarily true - that a “trend” towards significance does not necessarily imply significance will be achieved with greater statistical power? Furthermore, is there anyway to quantify the likelihood of obtaining significant results with higher power?

  • 1
    $\begingroup$ I'm curious if the MD felt that the difference in outcome would be 'clinically significant'. If an increase from 2.68% to 4.98% would lead to a meaningful difference in clinical outcomes then the MD is correct and it would be worth determining the difference more accurately. If that magnitude of difference is nowhere near big enough then it would be more difficult to justify additional experiments. $\endgroup$
    – ReneBt
    Sep 20 at 10:42

I am a bit surprised that this question did not get more traction ... So, I start the discussion and let's see if it moves further.

It is clear that the small sample size of your experimental group does not help to detect a real effect (if any). To illustrate it, I simulated a bigger experimental sample size but kept the same proportion (i.e. $n_2$ increases but $\hat p_2$ does not change). You did not mention the test statistic that you were using, so I assume that you were performing a "standard" test for two binomial proportions:

$$z=\frac{\hat p_1-\hat p_2}{\sqrt{\hat p(1-\hat p)\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$$ where $\hat p=\frac{n_1\hat p_1+n_2\hat p_2}{n_1+n_2}$.

In this scenario, the hypotheses for this test are:

  • $H_0$: the proportions for the two populations are equal ($\hat p_1 = \hat p_2$)
  • $H_A$: the proportions for the two populations are not equal ($\hat p_1 \neq \hat p_2$)

So, from the charts below, we can see that as $n_2$ gets bigger, p-value gets lower and power gets bigger (I used bpower from Hmisc package to determine the power):

n1 <- 783; e1 <- 21; p1 <- e1/n1
m <- 1:10
n2 <- 261*m; e2 <- 13*m; p2 <- e2/n2

p <- (p1*n1 + p2*n2)/(n1+n2)
ts = (p1-p2)/sqrt(p*(1-p)*(1/n1 + 1/n2))

pvalue <- pnorm(ts)
power <- bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)
plot(n2, pvalue, type="l", col="red")
plot(n2, power, type="l", col="blue")

enter image description here enter image description here

For your current study, we have the following power:

n2 <- 261; e2 <- 13; p2 <- e2/n2
bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)

##     Power 
## 0.4495768

So, it has a 45% chance of detecting an effect that exists. But sample size is not the only parameter that impacts the power. Now let's assume that your experimental group has actually double observations with the same proportion (26 successes out of 522 trials):

n2 <- 522; e2 <- 26; p2 <- e2/n2
p <- (p1*n1 + p2*n2)/(n1+n2)
ts = (p1-p2)/sqrt(p*(1-p)*(1/n1 + 1/n2))

pvalue <- pnorm(ts)

## [1] 0.01450141

bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)

##    Power 
## 0.583719

By doing this, the power has increased (power = 58%) and therefore we increased our chance to detect an effect that actually exists. We are also in a position where we can reject the null hypothesis.

So, if we assume now that we have the same sample size ($n_2=522$) but instead of having 26 successes we have only 24. The entire test results are different: we are in a position where we cannot reject the null hypothesis and power significantly decreases (power = 46%).

n2 <- 522; e2 <- 24; p2 <- e2/n2
p <- (p1*n1 + p2*n2)/(n1+n2)
ts = (p1-p2)/sqrt(p*(1-p)*(1/n1 + 1/n2))


## [1] 0.03158036

bpower(p1, p2, n1 = n1, n2 = n2, alpha = 0.05)

##     Power 
## 0.4615091

Since power is not only linked to sample size, it would require to maintain the same "success rate" (or more) and a bigger sample (about $n_2 = 1700$ using bsamsize function) to be as affirmative as your colleague. Indeed, we have seen that 2 drops in successes out of 522 (0.4%) is sufficient to change the conclusion of the study.

  • 4
    $\begingroup$ The need to maintain the same success rate is a key observation. Adding more samples can result in less impressive results. Anything goes. If the current sample size cannot be extended then it's best to drop the arbitrary idea of "significance" and compute a compatibility (confidence) interval for the difference in probabilities. This interval helps to point out the limitation of (current) knowledge. $\endgroup$ Sep 20 at 11:33
  • $\begingroup$ It seems like to get correct syntax highlighting here you need to do ` ```r ` , that is, not put ` {r}` ... Strange. $\endgroup$ Oct 13 at 12:28
  • $\begingroup$ Good to know @kjetilbhalvorsen! It was a copy/paste from markdown... Thanks :-) I won't forget it next time :-) $\endgroup$
    – Pitouille
    Oct 13 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.