Power of lady tasting tea experiment

Question

In the famous Fisher's experiment the observable is the number of corrected guessed cup $k$ having two kind of cup $A$ and $B$. Usually it is interesting to compute the critical region to reject the null hypothesis (the lady is randomly guessing) given the size of the test $\alpha$. This is easily done using the hypergeometric distribution. In the same way I can compute the size of the test given the critical region.

A different question is: how to compute the power of the test, given an alternative hypothesis? Suppose for example that the lady is able to guess correctly with probability on the single cup $p=90\%$ ($P(\text{guess} A|\text{true} A)=P(\text{guess } B|\text{true } B)=0.9$). What is the power of test, assuming a total number of cups equal to $N=8$ and a total number of cups of one kind $n=N/2=4$? (Unfortunately) the lady knows $n$.

Said in other words: what is the distribution of $k=$(number of correct cups under the alternative hypothesis) if the lady knows that there are $n$ cup of one kind?

Answer 1

The distribution of the correct number of guesses under the alternative hypothesis follows a non-central hypergeometric distribution, which is parameterized in terms of the odds ratio, that is, how much higher are the odds that the lady will guess "tea first" when in fact tea was really added first as opposed to when in fact milk was added first (or the other way around). If the odds ratio is 1, then we get the central hypergeometric distribution.

Let's see if this works. I will use R for illustration purposes, using the MCMCpack package, which has function dnoncenhypergeom() for computing the density of a (non-central) hypergeometric distribution. It has arguments x for the correct number of guesses (careful: this is the correct number of guesses under one of the two conditions, for example, when tea was really added first), arguments n1, n2, and m1 for three of the four margins, and psi for the true odds ratio. Let's compute the density for x equal to 0 to 4 (with all margins equal to 4) when the true odds ratio is 1:

install.packages("MCMCpack")
library(MCMCpack)
sapply(0:4, function(x) dnoncenhypergeom(x, n1=4, n2=4, m1=4, psi=1))

This yields:

[1] 0.01428571 0.22857143 0.51428571 0.22857143 0.01428571

So, there is a 1.43% chance that the lady will make 8 correct guesses (i.e., she guesses all 4 cups correctly where tea was added first and hence she also guesses all 4 cups correctly where milk was added first) under the null hypothesis. This is in fact the amount of evidence that Fisher considered sufficient to reject the null hypothesis.

The probabilities specified in the question can be used to compute the odds ratio, namely, $(.90/(1-.90)) / (.10/(1-.10)) = 81$ (i.e., $\text{odds}(\text{guess}A|\text{true}A) / \text{odds}(\text{guess}A|\text{true}B)$). What are the chances now that the lady will guess all 8 cups correctly (i.e., she will guess all 4 cups correctly where tea was added first and hence also the 4 cups correctly where milk was added first)?

dnoncenhypergeom(4, n1=4, n2=4, m1=4, psi=81)

This yields:

[1] 0.8312221

So power is about 83% then.

Answer 2

Under the alternative the lady is not randomly guessing, but "not randomly guessing" covers an infinity of different situations. She might always guess perfectly or she might only do very slightly better than random guessing ... and in the general case there's not even a single-variable "scale" not-random to work along (so we don't even have a power curve unless we restrict the kinds of non-random responses she might give).

So in order to compute a power, we must be very specific about how it's non-random (and just how non-random it is in that particular fashion).

We could suppose, for example, that she gets a sensation of how much each cup tastes like the milk was added first -- a "milk-firstiness" index which is a random variable on $(-\infty,\infty)$ that has a different (higher) mean when the milk is added first -- e.g. we might suppose that it's say normal or logistic, with mean $\mu_0$ and variance $\sigma^2=1/\omega^2$ ($\omega^2$ is known as "precision") when milk is added last and mean $\mu_1$ and variance $\sigma^2$ when milk is added first (indeed, a simpler but more restrictive presumption might be to set, say, $\mu_1=-\mu_0=1$ so that everything is now a function of one variable, the precision). So for any given values of those parameters, we could compute the probability that she gets all 8 cups correct (that the four smallest "milk-firstiness" values she experiences are associated with the four milk-second cups); if the exact calculation was too hard for us we could simulate it to any desired accuracy. [In the case where the nonrandomness is presumed to be a function of just one variable, we would have a power-curve -- a value for power for each value of the parameter.]

That's one specific kind of model for how she might perform "better than random" with which we might specify parameters and obtain a value for power.

We could of course suppose many other forms of non-randomness than this.