1
$\begingroup$

I want to compute the sample size to compare two groups for a binary outcome where we expect rare events. I will do an example with R. Assume the following expected probabilities of observing an event in both groups.

p1 <- 0.0001 ## 1 event out of 10000 subjects
p2 <- 0.05 ## 500 events out of 10000 subjects

Sample size calculation yields

power.prop.test(p1=p1, p2=p2, power=0.8, sig.level=0.05) ## n = 153 by group

Now I just try to compute the power with simulations using an exact test given the sample size.

set.seed(123)
R <- 10000 ## number of repetitions
n <- 158 ## I slightly increase the sample size because of the exact test
pval <- NULL
n.events1 <- n.events2 <- NULL
for (i in 1:R){

    x <- sum(rbinom(n, 1, p1))
    y <- sum(rbinom(n, 1, p2))
    ct <- matrix(c(n-x, x, n-y, y), nrow = 2)
    pval[i] <- fisher.test(ct)$p.val
    n.events1[i] <- x
    n.events2[i] <- y
}
mean(pval<0.05) ## = 0.799; fine we have 80% power

table(n.events1) ## 9833 times we have 0 event, 167 times 1 event only
p1*n ## = 0.0158 = expected number of events observed in group 1. So much less than 1...
mean(n.events1) ## = 0.0167 fine quite close to p1*n

table(n.events2) ## not that important for group 2
p2*n ## = 7.9 = expected number of events in group 2
mean(n.events2) ## = 7.86 fine

So both methods approximately match.

My issue is that if I decide to start a study with 158 subjects by group, it is highly likely that in group 1, I will not observe any event. I mean, out of 10000 repetitions, only 167 times we got 1 event. I am wondering if in the end it would be possible to analyze these data after completion. Also even if I'm lucky enough to observe 1 event in group 1.

Am I missing something? Is the approach a complete nonsense in this setting? Is the Fisher test not applicable? Or something else?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Strangely, power.prop.test gives you the power for the Pearson $\chi^2$ test of proportions, and yet you have used the Fisher Exact Test to calculate a p-value. Of course you can "analyze these data" if there are no events (or just 1 event) in group 1. The Fisher Exact Test does not require an evaluation of the information under the alternative hypothesis, which is why your simulation has not thrown errors (hey, is that why you didn't end up using prop.test instead?). Alternately, you can perform prop.test instead of fisher.test by adding 1 to your matrix ct and using a biased, but efficient test.

Improve this answer
$\endgroup$
6
  • $\begingroup$ True, the simulation did not throw errors... So yes, could be analyzed this way. I used the exact test, maybe because we're concerned by rare events, it could be more precise, but without any rationale... $\endgroup$
    – user3631369
    Commented Feb 16, 2021 at 18:02
  • $\begingroup$ I also had in mind to use other methods, like logistic regression for the analysis. Or Poisson regression (for estimating prevalence ratio instead of odds ratio). And we can read here and there that there is this 1:10 event rule of thumb for the analyis. Even though it could be randomized there's always could be some request to adjust for an extra variable. $\endgroup$
    – user3631369
    Commented Feb 16, 2021 at 18:04
  • $\begingroup$ Actually my initial problem was for an non-inferiority test. So in both groups we expect rare events, like in group 1 in the example I gave. But with the non-inferiority margin we end up with someting similar to the above. Would it be an issue in this case? Where in both groups we have quite rare events? Thanks $\endgroup$
    – user3631369
    Commented Feb 16, 2021 at 18:12
  • $\begingroup$ You're barking up the wrong tree. The Pearson test is just the score test of the logistic regression model, so empty cells are going to give you the same grief. Push back on adjustment/stratification in randomized design. If the KOLs demand it, use an appropriate simulation-based method to find the $N$ for a target power. When they see N=3,000+, they're easily discouraged. Even if they do, the +1 adjustment is recommended. Poisson still requires big Ns to get that desired precision. $\endgroup$
    – AdamO
    Commented Feb 16, 2021 at 19:05
  • $\begingroup$ It sounds like your idea is not well developed and you need to focus your ask here, offer or select an answer, then ask another question when you better understand. $\endgroup$
    – AdamO
    Commented Feb 16, 2021 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.