When and how to avoid inappropriate use of Fisher's exact test

Question

Could someone explain why Richard McElreath says Fisher's exact test is rarely used appropriately in his excellent Bayesian introduction book (Statistical Rethinking)?

For reference, the context is below:

Why aren’t the tests enough for innovative research? The classical procedures of introductory statistics tend to be inflexible and fragile. By inflexible, I mean that they have very limited ways to adapt to unique research contexts. By fragile, I mean that they fail in unpredictable ways when applied to new contexts. This matters, because at the boundaries of most sciences, it is hardly ever clear which procedure is appropriate. None of the traditional golems has been evaluated in novel research settings, and so it can be hard to choose one and then to understand how it behaves. A good example is Fisher’s exact test, which applies (exactly) to an extremely narrow empirical context, but is regularly used whenever cell counts are small. I have personally read hundreds of uses of Fisher’s exact test in scientific journals, but aside from Fisher’s original use of it, I have never seen it used appropriately. Even a procedure like ordinary linear regression, which is quite flexible in many ways, being able to encode a large diversity of interesting hypotheses, is sometimes fragile. For example, if there is substantial measurement error on prediction variables, then the procedure can fail in spectacular ways. But more importantly, it is nearly always possible to do better than ordinary linear regression, largely because of a phenomenon known as overfitting.

Answer 1

It's hard to read this quotation & not surmise that the author considers it a mere blunder to use Fisher's Exact Test when the marginal totals of a contingency table are not fixed by design. "Fisher's original use" of the test must refer to the famous lady tasting tea who "has been told in advance of what the test will consist, namely that she will be asked to taste eight cups, that these shall be four of each kind, [...]" (Fisher (1935), The Design of Experiments);^† & then "an extremely narrow empirical context" parses as "a sampling scheme applicable to few studies carried out in practice".

But it's not a blunder: conditioning on the sufficient statistic for the distribution of the data under the null hypothesis is a standard technique to eliminate nuisance parameters & come up with tests of the correct size (that's the basis of permutation tests). The marginal totals contain very little information which you can use to estimate the parameter of interest, the odds ratio; & rather a lot about the precision with which you can estimate it: the argument is that the sample space obtained by conditioning on both is much more relevant for inference than that obtained by conditioning on one only, or on the total count only. It is a horribly coarse sample space, however, resulting in a lamentable loss of power. How should relevance of the sample space be balanced against information loss? How much coarsening of the sample space is acceptable before an asymptotically valid or an unconditional test is preferred? These are vexed questions, & the analysis of two-by-two contingency tables has been controversial for half a century or more.

Given that this comes from a Bayesian text, I think the author's missed an opportunity to poke fun at the dilemmas a commitment to the use of frequentist methods can lead to—like Jaynes does in Probability Theory: The Logic of Science

† In a paper published the same year as his book, he used an example in which, though the sampling scheme is not explicitly given, at most one margin could have been fixed in advance, & most likely just the total count was fixed. Like-sex twins of convicted criminals are categorized as monozygotic vs dizygotic & as convicted of crimes themselves vs not convicted in a two-by-two table (Fisher (1935), "The Logic of Inductive inference", JRSS, 98, 1, pp 39–82). [Edit: The data come from Lange (1929), Verbrechen als Schicksal: Studien am kriminellen Zwillingen. Wetzell (2000), Inventing the Criminal: A History of German Criminology, 1880–1945, p 162] describes Lange's data collection procedure; it's indeed the total count that was fixed by the study design.]