Testing for equal proportions when sample sizes are very small Suppose I observe binary data for two samples (hopefully the notation below is obvious) and I wish to test the hypotheses:
$$H_0: p_1 = p_2$$
$$H_A: p_1 \neq p_2$$
I know there is a $z$-test for doing this but such tests are asymptotic tests. What would we do if the sample sizes were potentially very small (let's say $n_1 = 7$ and $n_2 = 15$)? Using CLT-based methods seems inappropriate here. For extra complication, $p_1$ and $p_2$ are believed to be small (maybe less than .1, or even .05). What is the most proper way to handle this? Is it the likelihood ratio test? That seems hard since I do not know what the common probability is if $H_0$ is true. Is it Wilson's score method anyway even though the sample sizes are not large and the parameters are close to 0?
 A: It turns out that you can use the worst-case exact distribution of the score (Pearson $X^2$) or Wald ($Z$)
As you note, the distribution under $p_1=p_2=p$ has an unknown parameter. If you fix the unknown parameter at some value $p_\mathrm{maybe}$, there is no longer an unknown parameter and you can compute the exact null sampling distribution and the p-value.  Do this for a grid of values of $p_\mathrm{maybe}$ and take the largest p-value. As the p-value under the correct $p_\mathrm{maybe}$ will be correct, the maximum will be conservative (at worst).
If you do this for the likelihood ratio test, it doesn't work. The maximum p-value is basically 1. It does work for the score and Wald tests.  For the score test it gives you Barnard's conditional binomial exact test.
These two tests are more powerful than Fisher's exact test because their sampling distributions are less discrete. When the sample size is small they are usefully more powerful. The applet at this link shows the rejection regions for a bunch of tests include the Fisher test and these two exact tests.
A: You can use an exact test based on a hypergeometric distribution: Fisher's Exact Test. R statistical software has such a test fisher.test, which requires a $2 \times 2$ table of successes and failures.
Suppose you have $x_1 = 4$ successes from $n_1 = 16$ subjects for an estimate $\hat p_1 = .25$ from Group 1 and $n_2 = 12$ successes is $n_2 = 18$ for $\hat p_2 = 0.67.$ Are $\hat p_1$ and $\hat p_2$ significantly different at the 5% level? The table TBL and test are as follows. The P-value $0.02 < 0.05 = 5\%$ indicates that
the two groups gave significantly different results.
TAB = rbind(c(4, 12), c(12, 6));  TAB

     [,1] [,2]
[1,]    4   12
[2,]   12    6

fisher.test(TAB)


    Fisher's Exact Test for Count Data

data:  TAB
p-value = 0.02042
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.02802673 0.90758654
sample estimates:
odds ratio 
 0.1768846 

For sample sizes below 20 or so, the difference in observed proportions has
to be quite large, as here, in order to get a significant result at the 5% level.
An alternative test is a chi-squared test using the same kind of table. This is
an approximate test, especially for small counts. There is controversy whether
to use Yates' continuity correction (which tends to give P-values that are somewhat
too large) or not (P-values may be too small). The implementation of chisq.test in R can simulate a more reliable P-value (often nearly matching that of Fisher's Exact Test).
chisq.test(TAB, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  TAB
X-squared = 5.9028, df = NA, p-value = 0.02249

Both tests will accept $2 \times k$ tables in case you want to begin by
looking for differences among $k$ groups. [If $k$ is too large, the simulations
involved may overwhelm available computer memory customarily allocated to R
or your patience waiting for the result. (Without sim=T the chi-squared test
does not use simulation.)]
