1
$\begingroup$

Say I need to test two different product features ({existing/control: blue} vs {new/treatment: red} font on webpage, for example), and need to boil my analysis down a to a single go/don't go criteria for launching a new feature.

I'm oversimplifying, but a frequentist would set up an experiment and use p-values to decide if the new feature was significantly better than the control (and if the feature should be launched.)

Bayesian analysts, by contrast, can generate full posteriors over the efficacy of the control and treatment features. But it seems there's some debate of where to go from here. I've read that the Bayes Factor can produce a single number and its value can determine whether to launch the feature. However, I've read that Andrew Gelman isn't a huge enthusiast of the Bayes Factor as it's reductionist in nature.

(In this product feature context, assume a Beta-Binomial conjugate prior as the math is fairly friendly and the posterior is tractable w/o need of MCMC methods.)

In a YouTube Video - Easy as ABC: A Quick Introduction to Bayesian A/B Testing in Python (Will Barker) I watched, the author used Bayesian inference to estimate the control and treatment efficacy parameters, then used monte carlo simulations to estimate the overlap between distributions. He alleged that this number was analogous to p-values and could be used in conjunction with an alpha (say 0.05) to decide whether to launch the new feature.

I've also come across terms like BIC, AIC, and WAIC; I'm not sure if these are improvements upon the Bayes Factor or totally different metrics.

At any rate, I'd like to solicit the community's recommendations on launch/no-launch decisions using Bayesian inference: What are your thoughts?

$\endgroup$

1 Answer 1

1
+50
$\begingroup$

I answer this fairly thoroughly in What test/model would I use for A/B testing given 3 groups. I'll add some comments here for completeness.

In my opinion, you frame the problem correctly as a decision. Launching the change is a decision, and doing nothing is a decision. We should understand what we stand to lose under each decision.

A frequentist approach does not evaluate a decision per se. The p value from a frequentist analysis only tells us the probability of observing such a difference under the null hypothesis (assuming all the assumptions about the data are valid). The difference between control/exposure might be very small and yet the p value may indicate significance. Choosing to implement this change may fail to account for costs incurred in implementation. And for what? A small change in the metric? This seems short sighted to me.

In the link I provide, we get to specify what our loss function is. I use a very simple loss function (essentially, how much incremental we stand to lose were we to choose the wrong variant), but that is only for convenience. Bayesian decision making is superior (again, in my opinion) because we get to specify whatever loss applies to us.

$\endgroup$
2
  • $\begingroup$ Do you have any thoughts on Bayes factor vs other metrics? I see in your linked post you defined a custom loss function; I'll give this some thought as an alternative. $\endgroup$
    – jbuddy_13
    Aug 26, 2021 at 15:43
  • 2
    $\begingroup$ @jbuddy_13 Bayes factors are used to compare evidence between two models. In my own opinion, Bayes factors are very close to frequentism (comparing an alternative model against a sort of null model) in which case why not just opt for a traditional statistical approach? In these sorts of settings, its reasonable to assume the intervention has some non-zero effect, its just a question of precision of the estimate. However, I'm not an expert on Bayes factors, so I might be off base here. $\endgroup$ Aug 26, 2021 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.