5
$\begingroup$

If a one-sided Fisher's exact test testing the null hypothesis that A is not superior to B has a p-value of 0.98, is it true that the Fisher's exact test p-value for the null hypothesis that B is not superior to A is 1-0.98 = 0.02?

The question arises from the paper at nejm.org/doi/full/10.1056/NEJMoa1100403 where the primary outcome had a one-sided p-value of 0.98, but when I calculated the P value with a one-sided FET, I got a p-value of 0.09. Therefore, wondering why the "other" one-sided FET wouldn't have a p-value of 1-0.09 = 0.91 (instead of 0.98). Would appreciate help in knowing where I'm going wrong.

The Stata command used to generate the p-value of 0.09 was:

 csi  2  7 2850 2854 , exact

                 |   Exposed   Unexposed  |      Total
-----------------+------------------------+------------
           Cases |         2           7  |          9
        Noncases |      2850        2854  |       5704
-----------------+------------------------+------------
           Total |      2852        2861  |       5713
                 |                        |
            Risk |  .0007013    .0024467  |   .0015754
                 |                        |
                 |      Point estimate    |    [95% Conf. Interval]
                 |------------------------+------------------------
 Risk difference |        -.0017454       |   -.0037999    .0003091 
      Risk ratio |         .2866159       |    .0595926    1.378504 
 Prev. frac. ex. |         .7133841       |   -.3785042    .9404074 
 Prev. frac. pop |         .3561301       |
                 +-------------------------------------------------
                                  1-sided Fisher's exact P = 0.0904
                                  2-sided Fisher's exact P = 0.1793
Improve this question
$\endgroup$
1
  • $\begingroup$ The details on the particular table in question made a huge difference. $\endgroup$
    – Karl
    Commented Oct 2, 2011 at 3:48

2 Answers 2

Reset to default
6
$\begingroup$

The particular table helps a lot. The Fisher's exact test assigns probabilities to tables with these particular marginals using the hypergeometric distribution. In this case, we're thinking of drawing 9 balls from an urn (the cases) with 2852 white balls (exposed) and 2861 black balls (not exposed). The number of white balls drawn is the count for exposed cases. The distribution is:

0     1     2     3     4     5     6     7     8     9 
0.002 0.018 0.071 0.165 0.247 0.246 0.163 0.070 0.017 0.002

The one-sided test in your output is giving the probability of 2 or fewer: 

0.002 + 0.018 + 0.071 = 0.0904

The one-sided test in the other direction would give the probability of 2 or more, which is 1 minus the probability of 0 or 1:

1 - (0.002 + 0.018) = 0.98

Note that the two-sided test is the probability of 0, 1, 2, 7, 8, or 9, which does come to 0.179.

So the p-values for the two one-tailed tests don't add to one, because they each include the particular observed value and the distribution is discrete.

Improve this answer
$\endgroup$
0
1
$\begingroup$

Fisher's test has the null hypothesis that A is the same as B. Suppose that we have 50 observations each for A and B. Fisher's test puts all 100 observations in a bag, shakes it, pulls out 50, calling them A, then pulls out the remaining 50 and calls them B. Calculate the average of A - B. Repeat this permutation many times to get a distribution of the averages of A - B.

Since our alternative is that A is superior to B, we reject if we got a really high value for A - B in the true sample. We count how many permutations were bigger than the one from the actual sample. You said 98% were. This means that A - B from the true sample is really low---only 2% of the permutations were lower.

If the alternative hypothesis was that B is superior to A, then we are hypothesizing that A - B is small; this would have a p-value of 0.02---only 2% of the permutations were smaller. We could reject the null that A is the same as B.

I caution against forming hypotheses after you see the data, however, as this biases your test statistics.

Improve this answer
$\endgroup$
4
  • $\begingroup$ here you seem to be describing a permutation test, but the question concerns "Fisher's exact test", though in that case the "A superior to B" part is confusing. $\endgroup$
    – Karl
    Commented Oct 1, 2011 at 19:09
  • $\begingroup$ A permutation test is one way to implement Fisher's exact test and I think that it's the easiest way to think about just what hypothesis that it's testing (i.e., that all the observations come from the same bag and that assignment to groups is just an arbitrary label). You could also use formulas to calculate the p-values directly without needing to perform the permutations, but I don't think that is as clear. $\endgroup$
    – Charlie
    Commented Oct 1, 2011 at 20:56
  • $\begingroup$ yes, Fisher's exact test is a type of permutation test, but what does "A superior to B" have to do with Fisher's exact test? $\endgroup$
    – Karl
    Commented Oct 1, 2011 at 22:25
  • $\begingroup$ Ah, based upon the original question, I assumed that A was the outcome under control, say, and B was the outcome under the treatment regime. The update to the question suggests that this is not what the OP had in mind. $\endgroup$
    – Charlie
    Commented Oct 2, 2011 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.