8
$\begingroup$

I want to evaluate the difference in proportions in declined applications and exceptions from policy from one quarter of data to the next. My sample sizes are generally not small (several hundred to several thousand), but the proportions may range anywhere from under 1% up to 60% depending on the specific set of data. As the two compared proportions diverge, using either test seems to produce similar p-values. However, if the proportions are relatively close, the z-test and Fisher's test p-values disagree more. Overall, the significance under either test tends to agree. Is one test more appropriate in this scenario based on the my data and any implied assumptions?

My current research seems to suggest that Fisher's Exact Test is always better since it is an exact test and a z-test an asymptotic test. Is this correct? If this is the case, why would anyone with the appropriate computational power ever conduct a z-test for proportions over Fisher's Exact Test?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ The Fisher exact test is using a different assumption. It assumes that the marginals are both fixed a priori. The z-test is more a counterpart of the binomial distribution and Bernard's test. You could see it as a Wald test. $\endgroup$
    – Sextus Empiricus
    Commented Mar 1, 2023 at 20:58

1 Answer 1

Reset to default
0
$\begingroup$

Fisher's exact test offers a more exact p-value by running every possible scenario with the given data set and figures out the total number of possible successes and total number of possible failures at the given sample size, then it converts those totals into a p-value. Also note that the total possible outcomes goes up by a factorial.

The normal approximation can be computed by a calculator and is much easier to compute as the sample size gets large. The normal approximation also becomes closer to fisher's exact test as the sample size gets large.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This has been discussed extensively on this site. P-values from Fisher's "exact" test are not very accurate. Some people use a continuity correction on Pearson's $\chi^2$ to make it more like Fisher's test. Not a good idea. $\endgroup$
    – Frank Harrell
    Commented Jun 29 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.