5
$\begingroup$

Let's say we are running an A/B testing and each data point has a binary response. We would like to test whether the ratio of true are different between A and B. (e.g. ask a yes/no question to both group A and group B and would like to test if there is difference in the ratio of "Yes" between the two groups)

I understand I can apply z-test if we can approximate the distribution of the number of true data (modeled as binomial distribution) as normal distribution, but there are cases that we can not approximate the binomial distribution as normal distribution.

So my question is, is there any statistical test available for given two binomial distributions $A \sim \mathrm{Bin}(n, p_a)$ and $B \sim \mathrm{Bin}(m, p_b)$ where $n$ and $m$ are the sample size of A and B to test if $p_a$ and $p_b$ are different without approximation to normal/Poisson distribution?

$\endgroup$
  • 3
    $\begingroup$ Fishers test or Chi-square test? $\endgroup$ – user2974951 Jun 8 '20 at 12:58
  • 2
    $\begingroup$ A Chi-square test still makes a distributional assumption. It assumes the sample size is large enough that the test statistic is Chi-square distributed. It's the same problem as assuming a normal approximation. I agree that Fishers test is appropriate though. $\endgroup$ – Eli Jun 8 '20 at 13:47
6
$\begingroup$

You can just use a Fisher Exact Test. Let us know if you have trouble following what it does.

Not super related, but if you're thinking of difference of binomials, it's nice to convince yourself that if $p_1 \neq p_2$, then the difference is not itself a binomial! I think that's kinda fun to think about.

$\endgroup$
4
$\begingroup$

So my question is, is there any statistical test available for given two binomial distributions $A \sim \mathrm{Bin}(n, p_a)$ and $B \sim \mathrm{Bin}(m, p_b)$ where $n$ and $m$ are the sample size of A and B to test if $p_a$ and $p_b$ are different without approximation to normal/Poisson distribution?

One way to do this is with Fisher's exact test. But, Fisher's exact test is often not 'so exact' because it is conditioning on the total number of (marginal) cases (considering this is an experimentally fixed number) which is often not the case (The 'exact' in the name for this test refers to the exact computation instead of an approximation. But in practice, it is often a computation for the wrong, non-exact, question).

The alternative is using Barnard's test, which considers a range of null hypotheses $A \sim B \sim \mathrm{Bin}(n, p)$ where $p$ is an unknown (nuisance) parameter, and the test is done by selecting the worst case (highest p-value) out of all possible $p$.

$\endgroup$
  • 1
    $\begingroup$ Personally I'd say that these cases (when Fisher's exact test or Barnard's test are making a big difference) are more a sort of statistical mathematics. In practice, you either use a large sample size such that the z-test is sufficient, or you have some background information such that you use a Bayesian technique. In other cases statistics will just be a mathematical excuse for low quality of data and information. $\endgroup$ – Sextus Empiricus Jun 8 '20 at 15:11
  • $\begingroup$ Bayesian approaches (e.g. Bayesian logistic regression - or the super simple sampling p1 ~ Beta(a+successes in group 1, b+failures in group1) and p2 ~ Beta(c+success in group 2, d+failures in group 2) for suitable a,b,c and d that represent prior beliefs, and then looking at the sampled values of p1-p2 or p1/p2 or whatever else you like) indeed seem like a good suggestion. $\endgroup$ – Björn Jun 8 '20 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.