# Fisher's tea tasting, binomial exact test

Please see the famous Fisher's experiment on biologist B. Muriel Bristol-Roach's ability to discern taste in red tea here (see Lady Tasting Tea).

In this experiment Fisher gave Bristol-Roach 8 cups of tea, of which 4 are made by first adding tea to the cup, and the other 4 made by first adding milk to the cup. Bristol-Roach remarkably correctly selected all 4 cups prepared by the same method. Then Fisher quantified the probability of her doing so by chance, and concluded that it was too small for her to do so just by chance.

I am wondering if a different method can be used, using the exact binomial test here, with $H_0$: the success rate = 0.5

Would this be enough to conclude that Bristol-Roach does have the ability to distinguish the teas, if the binomial exact test successfully reject $H_0$?

• To add to the answer by @brumar: I agree that the process outlined by brumar can result in a binomial test with p = 0.5. I wonder, however, if the coin toss to decide the method of preparation for each cup is necessary. If the numbers of milk-first and tea-first cups were predetermined, but the lady does not know the numbers, can't we still do a binomial test with p = 0.5? See the same issue discussed here: stats.stackexchange.com/questions/136584/… Aug 31 '17 at 21:18

This is a good idea, but the lady knows that there are 4 cup of tea for each type. This is a valuable information for the lady, which makes things wrong if we model the process via a binomial distribution. The problem is that the variables (successes at each trial) you want to consider are not independent and identically distributed.

I think you have thought to model the process by at least one of these cases:

Case 1: You study the number of success among the 4 selected cups.
Under this representation the statistic is 4 success over 4 trials. Under the null, each one would have a probability of 0.5 to be milk-first. This is mathematically right, but these probabilities are not independents.
Illustration: If the cup A,B and C are mistakes, there are good chances that the last one is a good one because among the 5 remaining cups, there are 4 milk-first cups remaining and only one milk-after cup.

Case 2: You study the number of success among the 8 presented cups.
Under this representation the statistic is 8 successes over 8 trials. This is the same problem of non independence.
Illustration: If she judged well the first 7 cups, the probability that she also judge well the last cup is 1. Because, relatively to the experimental setting, by elimination, there is no possibility that the lady is right about 7 cups and wrong about one.

In more mathematical term, for both cases, $\newcommand{\success}{\rm success}P(\success_i)$ is not independent with $P(\success_j)$.

Fisher avoided this problem by considering the selection process as a whole, enumerating the number of successful selections (well, only one) divided by the number of possible selections (4 amongst 8 = 70). Still, there is a simple raw formula which takes into account non independence, less beautiful than Fisher solution though:

\begin{align} P(\success) &= P(X_1=1)\times P(X_2=1|X_1=1)\times \\ &\quad\ \ P(X_3=1|X_1=1 \cap X_2=1)\times \\ &\quad\ \ P(X_4=1|X_1=1 \cap X_2=1 \cap X_3=1) \\ &= 4/8\times 3/7\times 2/6\times 1/5 \\ &= 1/70 \end{align}

A binomial test would be the correct answer to another kind of setting like this one I just made up.

• The judge toss a fair coin, if tails he prepares a milk-first tea, if head a milk-after tea. Obviously the lady does not know the result of the coin toss.
• The lady knows the process and will have to judge which kind of cup of tea was served.

With this setting, a binomial test, as you described it, with $H_0$: success rate = 0.5, would be undeniably a good approach.

• I see. I thought the lady did not know that there were exactly 4 cups of each type, and hence I was asking. The binomial test, though I really do not know much, does it not suffer from the discontinuous nature of the random variable? Also intuitively somehow I still think the success rate = 0.5 is easier to reject than what was used by Fisher here, though I cannot say why... Jul 5 '15 at 15:36
• Yes, it does suffer from it as the rejection region represents less than 5%. It suffers from conservativeness, especially for small samples. I don't get the second part of your comment as the Binomial test can't be used for the original lady tasting experiment. Jul 5 '15 at 16:16
• Hum. I realized that my current answer is not really good, I am editing it at the moment. Jul 5 '15 at 16:22
• +1, this is a nice answer. I tweaked the $\LaTeX$ for easier readability, & removed the spaces before the colons for more typical English punctuation use. If you don't like it, roll it back w/ my apologies. Jul 5 '15 at 19:10