The advantage of Bayesian A/B testing is that it considers our prior belief about the distribution. However, sometimes we do not have this, so we set prior to be uniform and work with visits and conversion counts. Why would one still use Bayesian A/B testing rather than a frequentist testing?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Why do you assume it's better in the first place? $\endgroup$– KodiologistCommented Jul 4, 2017 at 14:24
-
$\begingroup$ @Kodiologist I edited my question. However, I assume it is better for my question as "what is the probability that A is better than B". Bayesian can answer me that like " 60% sure that A is better than B", whereas a non-bayesian A/B testing can tell me that there is no difference between A and B. So, in fact, my question can be answered by a bayesian a/b testing. $\endgroup$– AlinaCommented Jul 4, 2017 at 15:08
-
$\begingroup$ And also, because I do not know the real effect size, so I do not know what should be the size of my groups to be in order to detect my unknown effect size. $\endgroup$– AlinaCommented Jul 4, 2017 at 15:14
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You seem to have answered the new formulation of your question yourself, in the comments. A primary appeal of Bayesian analysis of an experiment (which is what marketers call an A/B test) is that it lets you answer probabilistic questions about population values, such as "What is the probability that treatment A is better than treatment B?". This works even if you don't have rich prior information about the effects of A and B. (You still need to choose some prior distribution, though, and your choice between various so-called uninformative priors can be surprisingly consequential.) In a frequentist approach, by contrast, the true effects of the treatments are fixed, not random, so it makes no sense to ask about the probability that one is greater than the other.
answered Jul 4, 2017 at 18:08
$\begingroup$
$\endgroup$
I can think of two reasons that you might want to do this:
The posterior distribution $\mathbb{P}(\theta|D)$ gives us a detailed view of our knowledge about the conversation rate, $\theta$. It allows us to visualize our knowledge about the parameter's value, compare the similarity of two test variants, and use decision theory to estimate the expected value under each test.
If we are interested in maximizing our returns while running the test, we can use methods like Thompson sampling to acquire data about which test variant is better without the test becoming too expensive. This is not always the case, but for some expensive tests it may be preferable.
