5
$\begingroup$

Is binary hypothesis testing a better statistical term for what business intelligence often refers to as A/B testing? Wikipedia suggests that this is the term used withing the statistics community but Wikipedia is unreliable and there is no tag for such a term here. Since the statistical comparison of multi-grouped randomized experiments predates the term A/B test and the field of business intelligence I would suspect there to be a term. Is it simply "binary hypothesis testing" as suggested by wikipedia? Some historical context would also be appreciated.

$\endgroup$
5
$\begingroup$

The Wikipedia article has accurate information about A/B testing; binary hypothesis testing is another name for A/B testing. A/B testing and split testing are the most widely accepted terms in the business and marketing community. The exact origins of A/B testing are not well known but can be traced back to Google during the turn of the millennium. "Google engineers ran their first A/B test at the turn of the millennium to determine the optimum number of results to display on a search engine results page."

$\endgroup$
  • 1
    $\begingroup$ OK so I will use "binary hypothesis testing" if I want to be esoteric. It seems marketers have been doing "split testing" long before "A/B testing" was coined by google. The first such test I can find however was when William Sealy Gosset invented the t-test in 1908 for the Guinness brewery to compare beer. Would be interested if anybody can find something that predates this but it is rather poetic if it all started with beer. $\endgroup$ – Keith Jul 22 '14 at 18:45
  • $\begingroup$ opps seems Wikipedia was wrong. It has been updated. $\endgroup$ – Keith Jul 25 '14 at 13:00
1
$\begingroup$

"Binary hypothesis testing" is hypothesis testing when one wants to decide between two hypotheses.

"Two-sample hypothesis testing" is what is known colloquially as A/B testing.

"Paired hypothesis testing" when you compare the same sample before and after an event to find if it had an effect. Similar to A/B testing but not A/B testing.

$\endgroup$
0
$\begingroup$

I agree with the other very good answers. I think, it is mostly engineering backgrounds that prefer the term A/B test and the last years it has become a quite hot term especially within the context of web sites optimisation.

Have in mind that except A/B test you may encounter these terms:

  • A/B/C tests where you assess a control cohort against 2 alternative cohorts.
  • A/A tests where you empirically assess the quality and robustness of the statistical test against Type-I errors (the test should return that there's no difference in cohorts).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.