# Which kind of statistical exact test should I use?

I am testing a new kind of medicine against an older medicine in a very limited population. All the test subjects will first recieve the older, ineffective medicine. They will only recieve the new medicine if the old medicine was ineffective.

My data so far: None of 4 test subjects were cured by the old medicine, so all were given the new medicine. All 4 test subjects were cured by the new medicine. There are only two outcomes: cure or no cure.

I guess I cannot use a Fisher's exact test for this study, as the data are paired. Which test(s) could I use for this kind of experiment?

Peter Flom's answer makes a good point, but strictly speaking, the situation is more complicated than that. Treatment 2 is administered conditionally on the failure of treatment 1. Suppose the probability of success with treatment 1 is $p_1$, (respectively $p_2$ for treatment 2.)

Each patient who receives treatment 2 and is cured thereby contributes $(1-p_1)p_2$ to the likelihood. With 4 patients, that gives $(1-p_1)^4 p_2^4$.

You could then maximize the likelihood for $p_1$ and $p_2$ and compare, via a likelihood ratio test to the likelihood when $p_1=p_2$.

However, in your case, since no one was cured by the old treatment, MLE's for $p_1$ and $p_2$ are 0 and 1 respectively and that test will breakdown. But there is another way:

Assuming $p_1=p_2$, the MLE for the NULL model is $p=0.5$. The probability of getting what you got, or worse (only there isn't any worse) is then $0.5^8=0.004$.

However 4 patients is pretty small ... and from a design perspective, I would like to see a randomized trial. Or at least a cross-over. I don't know what the illness is, but if it were the common cold, that's exactly the result you would get if you applied two bogus treatments. The first would fail, and the patient would get better in time.

• (+1) I especially like the last paragraph. – MånsT Oct 4 '12 at 13:09

You could use some sort of permutation test, but with only 4 people there are only $2^4 = 16$ possible combinations. Yours is the most extreme one, so its p-value will be $\frac{1}{16} = 0.0625$ (This is a case where one can do the permutation test in one's head! That doesn't happen too often.)

It is impossible to even begin to interpret these data without knowing the natural history of the disease. As Placidia mentioned, if the disease is self-limiting (like a common cold), then these results are meaningless. The second treatment, whatever it is (even placebo), will seem to be effective. In contrast, if the disease is relentlessly progressive, these are impressive results that encourage you to set up a proper study.

It seems to me that this kind of study is worth doing to get a sense of whether the new drug works, as a prelude to a proper controlled study. I doubt that any kind of statistical analysis will prove useful.