5
$\begingroup$

Given a contingency table of the following rare event, why does my Fisher's exact test produce a "significant value" (i.e., < 0.05) but also show that the confidence interval for OR overlaps 1? My understanding is that a significant value should indicate an OR of 1 is excluded from the 95% CI. Is the p value one-sided and therefore needs to be interpreted as 2 x p (0.0573)?

        FALSE TRUE
  FALSE  3200    6
  TRUE    885    6

    Fisher's Exact Test for Count Data

data:  table
p-value = 0.02865
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
  0.9636962 13.5569691
sample estimates:
odds ratio 
  3.613991
Improve this question
$\endgroup$
1
6
$\begingroup$

This is an interesting phenomenon.

The difference is basically that the null hypothesis is more powerful because it ends up being one-sided. The confidence interval is not based on the same powerful tests (but it could).

Null hypothesis is tested with variable weight in left and right tails

Note that probability for the value in the cell 1,1 (which we call $x$) has the p-value of 0.028 only based on the left tail (the sum of probability for values 3200 and above). There is no right tail in this example, because the value can't get lower than 3194.

hypergeometric distribution

Confidence interval assumes a two-sided test with equal tails

The confidence interval is computed based on Fisher's noncentral hypergeometric distribution. The lower interval boundary is based on those values of the odds ratio for which the probability of observing $x \geq 3200$ is 0.025 or less.

This value happens to be below 1. This is not strange because we computed that the probability is 0.028 for the odds 1.

determining lower confidence boundary

The difference

Thus we can say: The difference is that the computation for the confidence interval is based on hypothesis testing with two tails with equal weight. But this is not the case for the hypothesis test that is used to compute the p-value.

The significance test for the null hypothesis will use the set values $\pm$ a certain distance from the maximum likelihood estimate. This might not need to be two equal tails.

In this example, the maximum likelihood estimate is at 3196. So the p-value is based on the probability that the observed $x \geq 3200$ or $x\leq 3192$. Due to the asymmetry this lower tail does not exist.

However, the computation for the confidence intervals uses a test with equal weight in both tails.

R-code

Below is some r-code that may help to manually compute the confidence intervals and p-values, which may be helpfull for gaining more insight in the fisher.test function. The code is a simplified version of the code that is under the hood of the fisher.test function.

### data
mat <- matrix(c(3200,6,885,6),2, byrow = T)

### parameters describing the data
x <- c(3194:3206)   ### possible values for cell 1,1
m <- 3200+6    ### sum of row 1
n <- 885+6     ### sum of row 2
k <- 3200+885  ### sum of column 1


### fisher test
test <- fisher.test(mat)
test

### manual computation of p-values
f <- dhyper(x,m,n,k)
plot(x,f)
pvalue <- sum(f[x >= 3200])

### compare p-values (gives the same)
pvalue
test$p.value


### non-central hypergemoetric distribution
### copied from fisher.test function in R
### greatly simplified for easier overview
logdc <- dhyper(x, m, n, k, log = TRUE)

### PDF
dnhyper <- function(ncp) {
  d <- logdc + log(ncp) * x
  d <- exp(d - max(d))        
  d / sum(d)
}


### CDF
pnhyper <- function(q, ncp = 1, uppertail = F) {
  if (uppertail) {
    sum(dnhyper(ncp)[x >= q])
  }
  else  {
    sum(dnhyper(ncp)[x <= q])
  }
}
pnhyper <- Vectorize(pnhyper)

### alpha level
alpha <- (1-0.95)/2

### compute upper and lower boundaries
x1 <- uniroot(function(t) pnhyper(3200, t) - alpha,
              c(0.5, 20))$root
x2 <- uniroot(function(t) pnhyper(3200, t, uppertail = T) - alpha,
              c(0.5, 20))$root

### plotting
t <- seq(0.2,20,0.001)
plot(t,pnhyper(3200,t, uppertail = T), log = "x", type = "l", xlim = c(0.20,20), ylab = "P(x => 3200)", xlab = "odds")
lines(c(10^-3,10^3), 0.025*c(1,1), col = 2)
lines(c(x2,x2),c(0,1), lty = 2)
Improve this answer
$\endgroup$
8
  • $\begingroup$ (+1) But I wouldn't say "the computation for the confidence interval assumes symmetry" - that seems to imply it's wrong when the test statistic doesn't have a symmetric distribution. The "equal-tail-probabilities" test is perfectly valid, & so are confidence intervals constructed by inverting it. Its rationale is that departures from the null in either direction are of equal interest (Cox, I think). Furthermore its p-value is guaranteed to be a bimonotone function of the null-hypothesis odds ratio (the corresponding confidence region is guaranteed not to be disjoint). $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Nov 6 '20 at 1:39
  • 1
    $\begingroup$ @Scortchi I have adjusted it. But I wouldn't say that the confidence interval used by fisher.test is a true coverage interval. It is valid in the sense that it guarantees to contain the true odds at least $\alpha\%$ of the time but it will not be exactly $\alpha\%$. $\endgroup$
    – Sextus Empiricus
    Nov 6 '20 at 6:00
  • $\begingroup$ actually, other intervals neither contain exactly $\alpha\%$ of the time the parameter because of the discreteness. But the interval from fisher.test can be made shorter in some cases and is inadmissible. $\endgroup$
    – Sextus Empiricus
    Nov 6 '20 at 6:34
  • $\begingroup$ Thank you! Indeed, discreteness implies that the true coverage of any "exact" interval is only equal to the nominal coverage for particular values. A direct way of shortening the "equal-tail-probabilities" CI is by inverting the test where, rather than doubling the smallest one-tailed p-value, you add on the larges achievable tail probability from the other tail that doesn't exceed it (Cox, Blaker, possibly others). But all the common alternative methods, shorter or not, can result in disjoint confidence regions - or the interval constructed may contain parameter values that ... $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Nov 6 '20 at 9:36
  • $\begingroup$ ... would be rejected by the corresponding test. (I recall reading, or at least glancing at, a paper that purported to show that all alternatives - subject to certain reasonable conditions - could lead to disjoint confidence regions - I'll try to dig it out.) $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Nov 6 '20 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.