2
$\begingroup$

I don't find the powerfulness of Fisher's exact test on Wikipedia. Are there any results about the power of FET vs other methods on 2x2 contingency tables?

$\endgroup$

2 Answers 2

3
$\begingroup$

No Fisher's Exact Test is not the most powerful because it doesn't condition on the sufficient statistic, the odds ratio, of the likelihood. Pearson's chi-square test, on the other hand, is the score test which is asymptotically equivalent to the likelihood ratio test, and the LRT is uniformly most powerful (among unbiased tests). You can even show that the $\alpha$-level Fisher's Exact test results in sub-alpha false positive rates (it does not attain its nominal size).

This has been known for a long time. Bennett and Hsu (1960) is one of many papers discussing the point. There are many reasons to like the FET, but power is not one of them.

$\endgroup$
5
  • $\begingroup$ What are the reasons to like FET? $\endgroup$ May 9, 2019 at 19:35
  • $\begingroup$ @user1424739 because there's no issue of small sample size approximation: it uses the exact distribution of the possible permutations of the contingency table. $\endgroup$
    – AdamO
    May 9, 2019 at 21:24
  • 1
    $\begingroup$ Under the same conditions as used in the FET (i.e. conditioning on the margins), we can compute exact p-values for distribution of the Pearson chi-squared and the LRT from the permutations (or we can simulate tables to get p-values to any desired degree of accuracy). However, the LRT is only guaranteed to be UMP for a one sided test. $\endgroup$
    – Glen_b
    May 10, 2019 at 2:21
  • $\begingroup$ @Glen_b: your point about LRT being UMP is only valid for a one-sided test is interesting. Is there a reference to this? Further, does it also apply to continuous distributions? It seems to me that your statement would only be true when the generalized likelihood ratio is asymmetric. $\endgroup$
    – user67724
    Nov 1, 2022 at 22:34
  • $\begingroup$ Likelihood ratios are usually not symmetric. They certainly aren't in this case. See the theorems on the connection between UMP and LRT, which are based on one-sided tests. e.g. see the Karlin-Rubin theorem section here en.wikipedia.org/wiki/… and the "Further Discussion" section, and the references. For two sided tests you need to add some further conditions. $\endgroup$
    – Glen_b
    Nov 1, 2022 at 22:35
1
$\begingroup$

A computation of the power always depends on your assumptions. I recently uploaded a preprint to arxiv about the implementation of an unconditional test for contingency tables tentatively called m-test (link). It is work in progress, but we used a Monte Carlo simulation to compare the power of the m-test, FET and Barnard's test in R at different values of $\alpha$. The results with different marginals suggest that Barnard's test is more powerful than FET in most cases. The m-test was consistently the most powerful test of these three. There is one example here (the figures are at the end of the file, the Barnard package is verbose). The algorithm for the m-test is packaged here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.