As this source states, one of the assumptions to perform Fisher's exact test of independence is that the row and column totals should be fixed. However, I find the explanation coming with it pretty vague and I couldn't find any examples illustrating the principle on the internet.
Would anyone have a good example explaining this assumption?
In my opinion, the source that you link to is wrong in that it is confusing conditioning with assumptions.
Fisher's exact test conditions on the margin totals, meaning that it does not use any information about independence that might be inferable from the margin totals. In probability theory, once you condition on a random variable, the random variable is then treated as fixed in the downstream computations. This however is purely a mathematical device. It is not at all the same as assuming that the margin totals were fixed in advance as part of the experimental design. It is simply a matter of choosing what information to extract from the data.
Conditioning actually reduces the assumptions made by the test. Since Fisher's test only uses the distribution of the cell counts given the margin counts, it therefore makes no assumptions about how the margin counts were generated.
The source you link to states that Fisher's exact test makes more assumptions than Pearson's chisquare test or the G-test. In my opinion, this statement is wrong. All three tests condition on the margin totals and all three make the same distributional assumptions. In the case of Fisher's test, the conditioning is explicit. For the other two tests, the conditioning is implicit from the way that the nuisance parameters are set but is nevertheless real. Fisher's exact test gives correct control of the type I error rate regardless of whether the marginal totals are fixed or not and the p-values are exact in the sense that they are derived from exact computations using the hypergeometric distribution.
IMO there is a lot of misunderstanding about this issue on the internet and it possibly derives from the question of optimality. If the margin totals are not fixed by the experimental design, then potentially there is information about independence contained in the row and column totals, and hence potentially one might be able to create a statistical test that is more powerful than Fisher's exact test. There is some interesting literature about this but in practice (i) there's not much information to retrieve if the counts are small and (ii) Fisher's exact test is already powerful enough when the counts are large.
Another misunderstanding is that people interpret "exact" to mean that the type I error rate is controlled exactly at a specified rate. Like all tests that return exact p-values for discrete counts, Fisher's test gives somewhat conservative control of any pre-specified type I error rate, simply because the p-values are granular (and this has nothing to do with whether the margins are fixed or not).
I am responding to Gordon Smyth's response, but it is too lengthy for just a comment.
I agree there are several issues with the Handbook of Biological Statistics cited and some pieces are simply inaccurate. I also agree that Fisher's test correctly controls for the type 1 error rate and doesn't assume both margins are fixed. This is one good reason to use Fisher's test over Pearson's chi-square test for smaller counts. However, compared to other exact tests, whether Fisher's test has fewer (or more) assumptions when performing the hypothesis test doesn't matter because an exact test means that the assumptions are met under the null hypothesis.
My disagreement is that I think there is much information to be gained by not conditioning on both margins. This can apply to both small and large datasets. A good practical example of why conditioning on both margins can yield problems is demonstrated from the following DOI article: 10.1016/j.jcrs.2018.10.044. The article randomized 3,640 eyes (large sample size) and found 1 event (endophthalmitis) in the treatment group and 7 events in the control group. They performed Fisher's exact test and reported a one-sided p-value of 0.035 (two-sided p-value was 0.070). The authors prespecified a one-sided test at alpha=0.05, which was used to derive the sample size, so they claimed the treatment was effective at reducing the event. However, a Letter to the Editor (DOI: 10.1016/j.jcrs.2019.03.026) argued a two-sided p-value should have been reported, or performed one-sided test at alpha=0.025, even if this was not prespecified. They essentially argue that one shouldn't test a two-sided tests at alpha=0.10 (even if this was used in the power calculation), so the study should have failed to reject the primary outcome and should not have claimed treatment is effective. While we can understand both sides, the problem is fixed if the authors simply used Barnard's exact test and did not condition on both margins. This would have yielded a two-sided p-value of 0.035 (cut the p-value in half!) and everyone would have been happy. There are absolutely no downsides with using Barnard's exact test over Fisher's exact test for a 2x2 table with zero or one margin fixed. IMO, Fisher's test should only be used for 2x2 tables when both margins are fixed and one is essentially assuming Fisher's test does not lead to insufficient power when they use it to analyze tables with one margin fixed.
Let's look at the difference in distribution of p-values for a Fisher's exact test when the marginals are fixed versus when the marginals are not fixed.
The setup: We have an urn with 20 red balls and 20 white balls. We draw two times a group of 20 balls from the urn and test the hypothesis whether the probability of drawing red and white balls is different between the two groups.
We can do these draws in two different ways, with and without replacement and the distribution of p-values from a Fisher test will be different
With the sampling without replacement (figure on the left) the marginals of a contingency table are fixed, the total number of red and white balls is allways 20 for each.
In that case the Fisher's exact test has the property that for every observable p-value we have that the cumulative distribution function is equal to the p-value value $F(p) = p$.
With the sampling with replacement (figure on the right) the number of red and white balls can very. The marginals are not fixed beforehand. And the hypothetical frequency of a p-value is not equal to it's nominal values. Although, due to it's conservative nature we do have that the frequency is guaranteed to be at least equal or lower $F(p) \leq p$.
So, if an experiment doesn't have the marginals pre-determined, then the real frequencies of a p-value does not equal the nominal frequency of a p-values.
Of course, the discrepancy doesn't mean that the Fisher exact test can not be used as an approximate approach. But it isn't always as exact as the name (unintentionally*) suggests, when the marginals are not fixed before the experiment. And also, there can be better tests such as Barnard's test which also has $F(p) \leq p$, but with more power.
set.seed(1)
balls = 20
sim = function() {
## sample two groups from the urn
b = 1:(2*balls)
y = sample(b, balls, replace = FALSE)
z = sample(b[-y], balls, replace = FALSE)
m = matrix(c(sum(y <= balls),
sum(y > balls),
sum(z <= balls),
sum(z > balls))
,2)
return(fisher.test(m)$p.value)
}
n =10^3
p = replicate(n,sim())
p = p[order(p)]
f = c(1:n)/n
plot(p,f ,
xlab = "observed p-value",
ylab = "empirical cumulative density distribution", type = "s", lwd ="2",
main = "without replacement")
lines(c(0,1),c(0,1), lty = 2)
set.seed(1)
balls = 20
sim2 = function() {
## sample two groups from the urn
b = 1:(2*balls)
y = sample(b, balls, replace = TRUE)
z = sample(b, balls, replace = TRUE)
m = matrix(c(sum(y <= balls),
sum(y > balls),
sum(z > balls),
sum(z <= balls))
,2)
return(fisher.test(m)$p.value)
}
n =10^3
p = replicate(n,sim2())
p = p[order(p)]
f = c(1:n)/n
plot(p,f ,
xlab = "observed p-value",
ylab = "empirical cumulative density distribution", type = "s", lwd ="2", main = "with replacement")
lines(c(0,1),c(0,1), lty = 2)
* The test is 'exact' because it depends on an exact distribution instead of an approximation. The test remains exact when the marginals are not fixed, but due to the conditioning on the marginals the nominal value of the p-values is not any more equal to the true cumulative frequency.
