2
$\begingroup$

I'd appreciate help checking my basic understanding of how to use Bayes factor in a practical A/B/C test.

Is this general understanding of how to use Bayes' factors in A/B/C testing correct?

Let's say:

  • I am testing 3 versions of a web page: A= an unchanged control version, B= new version 1 and C = new version 2.
  • I create an identical prior distribution based on historical data, to be used for all 3 versions.
  • Now I run my test, yielding a posterior distribution for each version.
  • I decide that the test has run long enough. At this point, I can create 3 bayes factors, one for each version, by using the respective posteriors and the common prior.
  • Other things equal, I can then select the test version with the highest BF10 value (where 1 is the version and 0 is the original prior).
  • There is the somewhat strange possibility that the control would win, and be quite different to the prior, which I guess would mean both that the control click-through performs best and that I had a poor initial understanding of what the historical click-through rate was.

Is the above correct?

I understand there are other ways to pick a winner; I'd just like to understand whether I have the use of bayes factor for verstion test winner selection correct. Thanks for any advice!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I think my understanding above was bogus. bayes' factors are calculated from priors, and I had misunderstood bayes factors as being what I now know are posterior distribution ratios. So I would say to me above: "no, your understanding was wrong."

I would delete this question altogether, but there seems to be a penalty for doing so.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Welcome to the site. It's actually a good question, no reason to delete it. There is plenty of opportunity for a better answer to come along, in part because I don't think "bayes' factors are calculated from priors" is fully correct or sufficient to explain it to future readers who might share your confusion. $\endgroup$
    – shadowtalker
    Commented Nov 8, 2023 at 21:32
  • $\begingroup$ Thank you @shadowtalker ... having a little better handle on this topic now, I can see that there are a number of "why use bayes factors? why not use posterior distribution ratios?" threads in SE. So my initial question seems redundant; I'll keep it around though if people think it helps. Viz: stats.stackexchange.com/questions/547114/…. and stats.stackexchange.com/questions/229852/… $\endgroup$
    – ouonomos
    Commented Nov 8, 2023 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.