6
$\begingroup$

Let's say we are running an A/B testing and each data point has a binary response. We would like to test whether the ratio of true are different between A and B. (e.g. ask a yes/no question to both group A and group B and would like to test if there is difference in the ratio of "Yes" between the two groups)

I understand I can apply z-test if we can approximate the distribution of the number of true data (modeled as binomial distribution) as normal distribution, but there are cases that we can not approximate the binomial distribution as normal distribution.

So my question is, is there any statistical test available for given two binomial distributions $A \sim \mathrm{Bin}(n, p_a)$ and $B \sim \mathrm{Bin}(m, p_b)$ where $n$ and $m$ are the sample size of A and B to test if $p_a$ and $p_b$ are different without approximation to normal/Poisson distribution?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Fishers test or Chi-square test? $\endgroup$
    – user2974951
    Commented Jun 8, 2020 at 12:58
  • 2
    $\begingroup$ A Chi-square test still makes a distributional assumption. It assumes the sample size is large enough that the test statistic is Chi-square distributed. It's the same problem as assuming a normal approximation. I agree that Fishers test is appropriate though. $\endgroup$
    – Eli
    Commented Jun 8, 2020 at 13:47

2 Answers 2

Reset to default
6
$\begingroup$

You can just use a Fisher Exact Test. Let us know if you have trouble following what it does.

Not super related, but if you're thinking of difference of binomials, it's nice to convince yourself that if $p_1 \neq p_2$, then the difference is not itself a binomial! I think that's kinda fun to think about.

Improve this answer
$\endgroup$
1
5
$\begingroup$

So my question is, is there any statistical test available for given two binomial distributions $A \sim \mathrm{Bin}(n, p_a)$ and $B \sim \mathrm{Bin}(m, p_b)$ where $n$ and $m$ are the sample size of A and B to test if $p_a$ and $p_b$ are different without approximation to normal/Poisson distribution?

One way to do this is with Fisher's exact test. But, Fisher's exact test is often not 'so exact' because it is conditioning on the total number of (marginal) cases (considering this is an experimentally fixed number) which is often not the case (The 'exact' in the name for this test refers to the exact computation instead of an approximation. But in practice, it is often a computation for the wrong, non-exact, question).

The alternative is using Barnard's test, which considers a range of null hypotheses $A \sim B \sim \mathrm{Bin}(n, p)$ where $p$ is an unknown (nuisance) parameter, and the test is done by selecting the worst case (highest p-value) out of all possible $p$.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Personally I'd say that these cases (when Fisher's exact test or Barnard's test are making a big difference) are more a sort of statistical mathematics. In practice, you either use a large sample size such that the z-test is sufficient, or you have some background information such that you use a Bayesian technique. In other cases statistics will just be a mathematical excuse for low quality of data and information. $\endgroup$
    – Sextus Empiricus
    Commented Jun 8, 2020 at 15:11
  • $\begingroup$ Bayesian approaches (e.g. Bayesian logistic regression - or the super simple sampling p1 ~ Beta(a+successes in group 1, b+failures in group1) and p2 ~ Beta(c+success in group 2, d+failures in group 2) for suitable a,b,c and d that represent prior beliefs, and then looking at the sampled values of p1-p2 or p1/p2 or whatever else you like) indeed seem like a good suggestion. $\endgroup$
    – Björn
    Commented Jun 8, 2020 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.