Testing for equal proportions when sample sizes are very small

Question

Suppose I observe binary data for two samples (hopefully the notation below is obvious) and I wish to test the hypotheses:

$$H_0: p_1 = p_2$$ $$H_A: p_1 \neq p_2$$

I know there is a $z$-test for doing this but such tests are asymptotic tests. What would we do if the sample sizes were potentially very small (let's say $n_1 = 7$ and $n_2 = 15$)? Using CLT-based methods seems inappropriate here. For extra complication, $p_1$ and $p_2$ are believed to be small (maybe less than .1, or even .05). What is the most proper way to handle this? Is it the likelihood ratio test? That seems hard since I do not know what the common probability is if $H_0$ is true. Is it Wilson's score method anyway even though the sample sizes are not large and the parameters are close to 0?

Answer 1

You can use an exact test based on a hypergeometric distribution: Fisher's Exact Test. R statistical software has such a test fisher.test, which requires a $2 \times 2$ table of successes and failures.

Suppose you have $x_1 = 4$ successes from $n_1 = 16$ subjects for an estimate $\hat p_1 = .25$ from Group 1 and $n_2 = 12$ successes is $n_2 = 18$ for $\hat p_2 = 0.67.$ Are $\hat p_1$ and $\hat p_2$ significantly different at the 5% level? The table TBL and test are as follows. The P-value $0.02 < 0.05 = 5\%$ indicates that the two groups gave significantly different results.

TAB = rbind(c(4, 12), c(12, 6));  TAB

     [,1] [,2]
[1,]    4   12
[2,]   12    6

fisher.test(TAB)


    Fisher's Exact Test for Count Data

data:  TAB
p-value = 0.02042
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.02802673 0.90758654
sample estimates:
odds ratio 
 0.1768846 

For sample sizes below 20 or so, the difference in observed proportions has to be quite large, as here, in order to get a significant result at the 5% level.

An alternative test is a chi-squared test using the same kind of table. This is an approximate test, especially for small counts. There is controversy whether to use Yates' continuity correction (which tends to give P-values that are somewhat too large) or not (P-values may be too small). The implementation of chisq.test in R can simulate a more reliable P-value (often nearly matching that of Fisher's Exact Test).

chisq.test(TAB, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 5.9028, df = NA, p-value = 0.02249

Both tests will accept $2 \times k$ tables in case you want to begin by looking for differences among $k$ groups. [If $k$ is too large, the simulations involved may overwhelm available computer memory customarily allocated to R or your patience waiting for the result. (Without sim=T the chi-squared test does not use simulation.)]

Answer 2

It turns out that you can use the worst-case exact distribution of the score (Pearson $X^2$) or Wald ($Z$)

As you note, the distribution under $p_1=p_2=p$ has an unknown parameter. If you fix the unknown parameter at some value $p_\mathrm{maybe}$, there is no longer an unknown parameter and you can compute the exact null sampling distribution and the p-value. Do this for a grid of values of $p_\mathrm{maybe}$ and take the largest p-value. As the p-value under the correct $p_\mathrm{maybe}$ will be correct, the maximum will be conservative (at worst).

If you do this for the likelihood ratio test, it doesn't work. The maximum p-value is basically 1. It does work for the score and Wald tests. For the score test it gives you Barnard's conditional binomial exact test.

These two tests are more powerful than Fisher's exact test because their sampling distributions are less discrete. When the sample size is small they are usefully more powerful. The applet at this link shows the rejection regions for a bunch of tests include the Fisher test and these two exact tests.