Suppose there are two samples $X_1\dots X_n$, $Y_1\dots Y_m$ of binary variables $X\sim Bern(p_1)$ and $Y\sim Bern(p_2)$. Is there any way to test the following hypothesis by the exact test?

$$H_0: p_1-p_2 = p; $$ $$H_1: p_1-p_2 \not= p,$$ where $p \not = 0$.

I do not want to use t-test since in my design $p_1$ and $p_2$ are small and I'm not sure about proper convergence of t-statistic.

  • $\begingroup$ How big, roughly, are $n$ and $m$? $\endgroup$
    – jbowman
    Apr 4, 2022 at 1:13
  • $\begingroup$ Does this help? $\endgroup$
    – Dave
    Apr 4, 2022 at 1:54
  • $\begingroup$ Would a simulation based approach fit ? $\endgroup$
    – Pohoua
    Apr 4, 2022 at 17:05
  • 2
    $\begingroup$ I don't think that an exact test can be derived, because the exact distribution of any test statistic that I can imagine (based on comparing $\hat p_1$ and $\hat p_2$ in some manner) will depend on the unknown true values of $p_1$ and $p_2$ in ways that for small samples (without normal approximation) are probably irregular and cannot be accounted for by standardisation. $\endgroup$ Apr 4, 2022 at 23:03
  • 1
    $\begingroup$ I recommend using the score test or the likelihood ratio test. There are closed-form estimators for $p_1$ and $p_2$ under the null hypothesis that their difference equals $p_1-p_2=p$. Although they are lengthy to write and involve cosines. These can be used to obtain a closed-form estimator of the score test statistic or likelihood ratio test statistic, but its exact distribution may be difficult to derive and the asymptotic $\chi^2_1$ approximation might be required. Maybe I will write an answer $\endgroup$
    – user277126
    Apr 7, 2022 at 3:09

1 Answer 1


I don't think there is an exact test in this case, but there is an approximate test. In general, concerns about the poor approximation of the approximate test are likely to be exaggerated unless you are exceedingly unfortunate to have both very very low $p_1$, $p_2$ and very very low $n$, $m$.

Certainly, higher $n$, $m$ will help, as the plot below shows. Note: I defined $p_2$=$p_1$+$p$.

enter image description here

Let x and y denote the numbers of successes observed in two independent sets of n and m Bernoulli trials, respectively, where $p_1$ and $p_2$ are the true success probabilities associated with each set of trials. Let $p_e=\frac{x+y}{n+m}$ and define:


$z$ is approximately ~ $Normal(0,1)$.

In your particular case, with m and n around 150, the approximation is very good as long as the smallest of the 2 probabilities is no less than ~ 0.04. I colored the sampling distribution in blue when $p_1>=0.04$ and in red otherwise.

enter image description here

#code for the first plot

for (i in 1:10000) {

plot(density(zhats),col="red",xlab="",main=paste0("n=",nvalue," m=",mvalue," p1=",p1," p=",p),lwd=2,
  • $\begingroup$ Thanks! I think I will. I wonder if no one has previously done a Monte Carlo analysis for all values $m,n,p_1,p_2$ to estimate the accuracy of the approximation in such a hypothesis. $\endgroup$
    – Kess
    Apr 6, 2022 at 9:49
  • 2
    $\begingroup$ Perhaps there is a summary of such a simulation somewhere. But having seen the results in this post and with the code in hand to try some more simulations, we are actually in position to make such conclusions ourselves. $\endgroup$ Apr 6, 2022 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.