One-sided Fisher's exact test and its complement

Question

If a one-sided Fisher's exact test testing the null hypothesis that A is not superior to B has a p-value of 0.98, is it true that the Fisher's exact test p-value for the null hypothesis that B is not superior to A is 1-0.98 = 0.02?

The question arises from the paper at nejm.org/doi/full/10.1056/NEJMoa1100403 where the primary outcome had a one-sided p-value of 0.98, but when I calculated the P value with a one-sided FET, I got a p-value of 0.09. Therefore, wondering why the "other" one-sided FET wouldn't have a p-value of 1-0.09 = 0.91 (instead of 0.98). Would appreciate help in knowing where I'm going wrong.

The Stata command used to generate the p-value of 0.09 was:

 csi  2  7 2850 2854 , exact

                 |   Exposed   Unexposed  |      Total
-----------------+------------------------+------------
           Cases |         2           7  |          9
        Noncases |      2850        2854  |       5704
-----------------+------------------------+------------
           Total |      2852        2861  |       5713
                 |                        |
            Risk |  .0007013    .0024467  |   .0015754
                 |                        |
                 |      Point estimate    |    [95% Conf. Interval]
                 |------------------------+------------------------
 Risk difference |        -.0017454       |   -.0037999    .0003091 
      Risk ratio |         .2866159       |    .0595926    1.378504 
 Prev. frac. ex. |         .7133841       |   -.3785042    .9404074 
 Prev. frac. pop |         .3561301       |
                 +-------------------------------------------------
                                  1-sided Fisher's exact P = 0.0904
                                  2-sided Fisher's exact P = 0.1793

Answer 1

The particular table helps a lot. The Fisher's exact test assigns probabilities to tables with these particular marginals using the hypergeometric distribution. In this case, we're thinking of drawing 9 balls from an urn (the cases) with 2852 white balls (exposed) and 2861 black balls (not exposed). The number of white balls drawn is the count for exposed cases. The distribution is:

0     1     2     3     4     5     6     7     8     9 
0.002 0.018 0.071 0.165 0.247 0.246 0.163 0.070 0.017 0.002

The one-sided test in your output is giving the probability of 2 or fewer:

0.002 + 0.018 + 0.071 = 0.0904

The one-sided test in the other direction would give the probability of 2 or more, which is 1 minus the probability of 0 or 1:

1 - (0.002 + 0.018) = 0.98

Note that the two-sided test is the probability of 0, 1, 2, 7, 8, or 9, which does come to 0.179.

So the p-values for the two one-tailed tests don't add to one, because they each include the particular observed value and the distribution is discrete.

Answer 2

Fisher's test has the null hypothesis that A is the same as B. Suppose that we have 50 observations each for A and B. Fisher's test puts all 100 observations in a bag, shakes it, pulls out 50, calling them A, then pulls out the remaining 50 and calls them B. Calculate the average of A - B. Repeat this permutation many times to get a distribution of the averages of A - B.

Since our alternative is that A is superior to B, we reject if we got a really high value for A - B in the true sample. We count how many permutations were bigger than the one from the actual sample. You said 98% were. This means that A - B from the true sample is really low---only 2% of the permutations were lower.

If the alternative hypothesis was that B is superior to A, then we are hypothesizing that A - B is small; this would have a p-value of 0.02---only 2% of the permutations were smaller. We could reject the null that A is the same as B.

I caution against forming hypotheses after you see the data, however, as this biases your test statistics.