I'm studying various statistical tests related to AB testing for businesses - Fisher's exact test, student's t-test, etc.

As I was doing so, I remembered the various convenient online AB testing calculators that let people punch in a few data points and output a p-value and "winner" version. For example -

enter image description here

There are so many of them. I wondered, what statistical tests are behind those calculators? Is there a more popular one among all of the calculators?



1 Answer 1


Both links you provided (Survey Monkey and AB Testguide) are just one-sided z-tests for a difference in proportions. Survey Monkey doesn't provide the formula, but you can easily verify it by trying the calculations yourself.

  • $\begingroup$ Thanks @Doctor Milt! Do you know why they don't use z score instead of t test here? I'm guessing in t test, you'd need to know the sample variances whereas in the z score calculation, you don't necessarily need variances? $\endgroup$
    – Peiran Yu
    Feb 3, 2023 at 5:05
  • $\begingroup$ It is a z-test rather than a t-test. The Central Limit Theorem tells us that the sample proportions are approximately Normally distributed when the sample sizes are large. $\endgroup$ Feb 7, 2023 at 11:01
  • $\begingroup$ sorry I meant the opposite... why don't they use t-test? Thanks for your help @Doctor Milt $\endgroup$
    – Peiran Yu
    Feb 8, 2023 at 13:07
  • $\begingroup$ The 2-sample t-test is appropriate when comparing the means of two normal populations with unknown (but equal) variances. In this example our populations are Bernoulli; each visitor becomes a conversion with probability $p$. We write $X_1,\ldots,X_n \sim \mathrm{Bernoulli}(p)$. We then use the fact that the distribution of $\bar{X}$ is approximately $ \mathrm{N}(p, p(1-p)/n)$ when $n$ is large to construct an (apprximate) z-test. $\endgroup$ Feb 8, 2023 at 22:24
  • 1
    $\begingroup$ Thank you, I didn't think of it from distribution's perspective before and i just spent some time to think about your comment. It pushed me to understand it much better. thanks for this, @Doctor Milt! $\endgroup$
    – Peiran Yu
    Feb 16, 2023 at 7:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.