Is binary hypothesis testing a better statistical term for what business intelligence often refers to as A/B testing? Wikipedia suggests that this is the term used withing the statistics community but Wikipedia is unreliable and there is no tag for such a term here. Since the statistical comparison of multi-grouped randomized experiments predates the term A/B test and the field of business intelligence I would suspect there to be a term. Is it simply "binary hypothesis testing" as suggested by wikipedia? Some historical context would also be appreciated.
The Wikipedia article has accurate information about A/B testing; binary hypothesis testing is another name for A/B testing. A/B testing and split testing are the most widely accepted terms in the business and marketing community. The exact origins of A/B testing are not well known but can be traced back to Google during the turn of the millennium. "Google engineers ran their first A/B test at the turn of the millennium to determine the optimum number of results to display on a search engine results page."
"Binary hypothesis testing" is hypothesis testing when one wants to decide between two hypotheses.
"Two-sample hypothesis testing" is what is known colloquially as A/B testing.
"Paired hypothesis testing" when you compare the same sample before and after an event to find if it had an effect. Similar to A/B testing but not A/B testing.
I agree with the other very good answers. I think, it is mostly engineering backgrounds that prefer the term A/B test and the last years it has become a quite hot term especially within the context of web sites optimisation.
Have in mind that except A/B test you may encounter these terms:
- A/B/C tests where you assess a control cohort against 2 alternative cohorts.
- A/A tests where you empirically assess the quality and robustness of the statistical test against Type-I errors (the test should return that there's no difference in cohorts).