0
$\begingroup$

The Question is at the end of the reasoning.

1 : Representation of binomial confidence intervals:

We want to know if the proportion of reds in zone 1 is significantly different from the proportion of reds in zone 2

df <- matrix(c(58, 51, 8, 17), nrow = 2,
                       dimnames =
                         list(c("zone1", "zone2"),
                              c("Red", "Blue")))
df


prop.table(df, margin=1)
df2<- data.frame(prop=c(0.8787879, 0.7500000),
                 zone=c(1,2))

df2

plot(prop~zone, data=df2, ylim=c(0,1))
segments(x0 = 1, y0 = binom.test(58, 66)$conf.int[1],
         x1 = 1, y1 = binom.test(58, 66)$conf.int[2], col = "red",
         lty = 1, lwd = 3)

segments(x0 = 2, y0 = binom.test(51, 68)$conf.int[1],
         x1 = 2, y1 = binom.test(51, 68)$conf.int[2], col = "red",
         lty = 1, lwd = 3)

abline(h=binom.test(58, 66)$conf.int[1], lty=2)
abline(h=binom.test(51, 68)$conf.int[2], lty=2)

Clearly, the two group's confidence intervals are overlapping. They should be far from statistical significance if we perform a test.

2: Some stats:

calculation of statistical power to use the fisher test:

library('statmod')

(power.fisher.test(p1=0.88, p2=0.75, n1=66, n2=68, alpha=0.08, nsim=1000, alternative="two.sided"))
power of 0.53 considering the 13% of difference and sample size in each group

Now the fisher test:

fisher.test(df)
p=0.07 which is close to a 0.05 significance

--> My question: Why the fisher test gives a p.value quite low (0.07), whereas the binomial confidence intervals are overlapping very much and the power calculation of the Fisher test in this context suggests to be very low (so we should not have so low pvalue) ?

Improve this question
$\endgroup$
10
  • 2
    $\begingroup$ Neither estimate is in the other group's 95% CI, so that indicates some evidence of a difference, but it's not strong evidence, since the CIs overlap. A p-value of 0.07 is consistent with some evidence of a difference, but not strong evidence. It happens by chance about 1 in 15 times when there is no difference at all. $\endgroup$
    – user2554330
    Jul 30, 2022 at 11:46
  • 1
    $\begingroup$ Also, bear in mind that Fisher's Exact Test (which is exact only in situations that rarely apply in practice) and the binomial test actually test different hypotheses. The hypotheses are closely related, but different nonetheless. Their p-values (and assocaited CIs) will be similar but need not be identical. Depending on the orientation of the table, the binomial test is actually closer to a Cochran-Mantel-Haenszel test for the equality of row (column) means. $\endgroup$
    – Limey
    Jul 30, 2022 at 11:59
  • 2
    $\begingroup$ This power calculation is called a posteriori power. Never, never do that. The a posteriori power is a decreasing function of the p-value., that makes no sense. $\endgroup$
    – Stéphane Laurent
    Jul 30, 2022 at 13:00
  • 1
    $\begingroup$ Thank you. Do you mean “a posteriori power” should never been calculated? Why checking overlap of 95% confidence intervals is a wrong way? $\endgroup$
    – SkyR
    Jul 30, 2022 at 16:52
  • 1
    $\begingroup$ Does this answer your question? Relation between confidence interval and testing statistical hypothesis for t-test Although that question is in the context of t-tests, the principle holds in all testing: lack of overlap of 95% confidence intervals is much more stringent than a p <0.05 test for a difference. $\endgroup$
    – EdM
    Jul 31, 2022 at 15:36

0

Reset to default