# On Fisher's exact test: What test would have been appropriate if the lady hadn't known the number of milk-first cups?

In the famous lady tasting tea experiment by RA Fisher, the lady is informed of how many milk-first/tea-first cups there are (4 for each out of 8 cups). This respects the fixed marginal total assumption of Fisher's exact test.

I was imagining doing this test with my friend, but the thought struck me. If the lady can really tell the difference between milk-first and tea-first cups, she should be able to figure out marginal totals of the milk-first/tea-first cups as well as which ones are which.

So here is the question: What test could have been used if RA Fisher hadn't informed the lady of the total number of milk-first and tea-first cups?

• Some would argue that even if the second margin is not fixed by design, it carries little information about the lady's ability to discriminate (i.e. it's approximately ancillary) & should be conditioned on. The exact unconditional test (first proposed by Barnard I think) is more complicated because you have to calculate the maximal p-value over all possible values of a nuisance parameter. Commented Feb 6, 2015 at 15:08
• In fact Barnard's test has a Wikipedia page. Commented Feb 6, 2015 at 15:11
• @Glen_b: Thanks. But the first point raises the very difficult question of how "nearly ancillary" a statistic has to be before you'd want to condition on it - how do you trade-off information loss against having a more relevant sample space? I don't have an answer. Commented Feb 6, 2015 at 17:40
• There's some discussion worth looking at (among both paper and discussants) in Yates, F. (1984) "Tests of Significance for 2 × 2 Contingency Tables", Journal of the Royal Statistical Society. Series A (General), Vol. 147, No. 3, pp. 426-463. Commented Sep 1, 2017 at 2:13
• Only on average, over 4 cups of tea with milk & 4 without, can we say she's a probability of $\frac{1}{2}$ of guessing correctly; & the distribution of her total of correct guesses is more concentrated about 4 than if it followed a binomial distribution. But if she guesses "milk" with $\frac{1}{2}$ probability, then she'll guess correctly with $\frac{1}{2}$ probability when there's milk in the tea & when there isn't, & the distribution of her total of correct guesses does follow a binomial distribution. So there is a nuisance parameter to consider, ... Commented Sep 2, 2017 at 14:30

One way of looking at this would be as a test for a common probability od 'success' across two binomial trials: the product-binomial model. Some would argue that even if the second margin is not fixed by design, it carries little information about the lady's ability to discriminate (i.e. it's approximately ancillary) & should be conditioned on. See When and how to avoid inappropriate use of Fisher's exact test for a discussion. Exact unconditional tests (introduced by Barnard (1947)) are more complicated because you have to calculate the maximal p-value over all possible values of a nuisance parameter, viz the common Bernoulli probability under the null hypothesis. More recently, maximizing the p-value over a confidence interval for the nuisance parameter has been proposed: see Berger (1996); exact tests having the correct size can be constructed using this idea.

Fisher's Exact Test also arises as a randomization test (in the sense of Edgington (2007)) : a random assignment of the experimental treatments allows the distribution of the test statistic over permutations of these assignments to be used to test the null hypothesis. In this approach the lady's verdicts are considered as fixed (& the marginal totals of milk-first and tea-first cups are of course preserved by permutation).

As in this particular experiment it would seem difficult to ensure the independence of trials required by the product-binomial model (without some kind of memory-wipe device to be used before the presentation of cups), F.E.T. as randomization test is likely the most sensible approach. As illustrated in Sextus' answer, the two-tailed p-value from Barnard's test if the lady correctly identifies all 4 milk-first & all 4 tea-first cups will be 0.4%. Now she didn't know beforehand there would be equal numbers of milk-first & tea-first cups; but what if she had just assumed that? A more appropriate p-value would then be the 2.9% from F.E.T. The experimenter could make a point of telling her not to assume even that not all cups are milk-first, or all cups tea-first; but how confident can we be that she'll not, consciously or subconsciously, let her verdict on one cup influence her verdict on another? Conversely, when the lady is told the totals of milk-first and tea-first cups, yet the totals of her verdicts are incompatible with these, F.E.T. is still appropriate.

Barnard (1947), "Significance Tests for 2 x 2 tables", Biometrika, 34, 1 – 2,

Berger (1996), "More Powerful Tests from Confidence Interval p-values", Am. Stat., 50, 4

Edgington & Onghena (2007), Randomization Tests

• Can Barnard::barnardw.test() be used here? What difference in computational complexity can be expected in practice? Commented Aug 21, 2015 at 9:28
• I'm not familiar with that package, but the help page you link to references exactly the test I was talking about. See also Exact. As to computational complexity, I don't know - it's going to depend on the maximization algorithm used. Commented Aug 21, 2015 at 9:39

Today, I read the first chapters of "The Design of Experiments" by RA Fisher, and one of the paragraph made me realize the fundamental flaw in my question.

That is, even if the lady can really tell the difference between milk-first and tea-first cups, I can never prove she has that ability "by any finite amount of experimentation". For this reason, as an experimenter, I should start with the assumption that she doesn't have an ability(null hypothesis) and try to disprove that. And the original experimental design/testing procedure (fixed margins, Fisher exact test) is a sufficient,efficient, and justifiable procedure to do so.

Here is the excerpt from "The Design of Experiments" by RA Fisher:

It might be argued that if an experiment can disprove the hypothesis that the subject possesses no sensory discrimination between two different sorts of object, it must therefore be able to prove the opposite hypothesis, that she can make some such discrimination. But this last hypothesis, however reasonable or true it may be, is ineligible as a null hypothesis to be tested by experiment, because it is inexact. If it were asserted that the subject would never be wrong in her judgments we hold again have an exact hypothesis, and it is easy to see that this hypothesis could be disproved by a single failure, but could never be proved by any finite amount of experimentation.

Barnard's test is used when the nuisance parameter is unknown under the null hypothesis.

However in the lady tasting test you could argue that the nuisance parameter can be set at 0.5 under the null hypothesis (the uninformed lady has 50% probability to correctly guess a cup).

Then the number of correct guesses, under the null hypothesis, becomes a binomial distribution: guessing 8 cups with 50% probability for each cup.

In other occasions you may not have this trivial 50% probability for the null hypothesis. And without fixed margins you may not know what that probability should be. In that case you need Barnard's test.

Even if you would do Barnard's test on the lady tasting tea test, it would become 50% anyway (if the outcome is all correct guesses) since the nuisance parameter with the highest p-value is 0.5 and would result in the trivial binomial test (it is actually the combination of two binomial tests one for the four milk first cups and one for the four tea first cups).

> library(Barnard)
> barnard.test(4,0,0,4)

Barnard's Unconditional Test

Treatment I Treatment II
Outcome I            4            0
Outcome II           0            4

Null hypothesis: Treatments have no effect on the outcomes
Score statistic = -2.82843
Nuisance parameter = 0.5 (One sided), 0.5 (Two sided)
P-value = 0.00390625 (One sided), 0.0078125 (Two sided)

> dbinom(8,8,0.5)
[1] 0.00390625

> dbinom(4,4,0.5)^2
[1] 0.00390625


Below is how it would go for a more complicated outcome (if not all guesses are correct e.g. 2 versus 4), then the counting of what is and what is not extreme becomes a bit more difficult

(Note as well that Barnard's test uses, in the case of a 4-2 result a nuisance parameter p=0.686 which you could argue is not correct, the p-value for 50% probability of answering 'tea first' would be 0.08203125. This becomes even smaller when you consider a different region, instead the one based on Wald's statistic, although defining the region is not so easy)

out <- rep(0,1000)
for (k in 1:1000) {
p <- k/1000
ps <- matrix(rep(0,25),5)   # probability for outcome i,j
ts <- matrix(rep(0,25),5)   # distance of outcome i,j (using wald statistic)
for (i in 0:4) {
for (j in 0:4) {
ps[i+1,j+1]  <- dbinom(i,4,p)*dbinom(j,4,p)
pt <- (i+j)/8
p1 <- i/4
p2 <- j/4
ts[i+1,j+1] <- (p2-p1)/sqrt(pt*(1-pt)*(0.25+0.25))
}
}
cases <- ts < ts[2+1,4+1]
cases[1,1] = TRUE
cases[5,5] = TRUE
ps
out[k] <- 1-sum(ps[cases])
}

> max(out)
[1] 0.08926748
> barnard.test(4,2,0,2)

Barnard's Unconditional Test

Treatment I Treatment II
Outcome I            4            2
Outcome II           0            2

Null hypothesis: Treatments have no effect on the outcomes
Score statistic = -1.63299
Nuisance parameter = 0.686 (One sided), 0.314 (Two sided)
P-value = 0.0892675 (One sided), 0.178535 (Two sided)

• (1) It's true that an uninformed lady must have a 50% probability of correctly guessing a cup (across equal numbers of 'milk first' & 'tea first' cups), but it doesn't follow that she has a 50% probability of guessing 'milk first'. A lady with a 68.6% probability of guessing 'milk first, regardless of the true value, is surely not a fantastical notion. (2) I added some discussion on the independent-trials assumption to my answer. Commented Feb 27 at 9:09