1
$\begingroup$

I'm trying to go through this article http://developers.lyst.com/data/2014/05/10/bayesian-ab-testing/

and I see that they choose a Beta(3, 50) prior and make an argument for that. However, if you use their calculator and choose a Beta(1, 1) prior, the results don't seem too different. In fact I've been playing around with both small and large sample sizes, and a Beta(1, 1) prior often times generates a higher probability that A > B. What are some downfalls of using the wrong prior. Does our probability estimate mean less?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

You can't really say a prior is wrong, because it is up to you to decide what the prior is. One comfort is this: under not too restricting conditions your prior does not matter (if you have enough data). This is Bernstein Von Mises theorem.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Yes but what are some potential consequences of an inaccurate prior in smaller sample sizes? False positives? $\endgroup$
    – John
    Commented Feb 6, 2015 at 19:32
  • 1
    $\begingroup$ @john I really dont know how to answer this question. I think a false positive is more of a frequentist term. I dont have anything against the frequentist paradigm but I dont see how one can give meaning to frequentist ideas in a Bayesian setting. $\endgroup$
    – Yair Daon
    Commented Feb 11, 2015 at 4:15
  • $\begingroup$ @YairDaon Priors can't be 'wrong' but they can be inaccurate, and that can lead to making the wrong inference. Asymptotically the posterior is independent of the choice of prior, but there aren't many cases in A/B testing where we can afford to collect arbitrarily large amounts of data. Choice of a poor prior means that you will need to collect more data to get to an accurate distribution. $\endgroup$
    – PrestonH
    Commented Dec 25, 2018 at 14:42
0
$\begingroup$

You could have unexpected issues arise with strong priors, especially with low, unequal sample sizes. For example, say, in an AB test, you specify priors for your conditions in which they both have the same mean and high precision (i.e., little variance in plausible parameter values). If you have, say, lots of data for your group A (control group) and little data for your group B (test group), group A will be able to overcome the strong prior while group B may not be able to overcome the strong prior. This could be an issue when say both the means of the data for group A and B are higher than the prior mean. There will be enough data in group A for its posterior distribution to shift away from the prior but not enough for group B's to do so. So you could end up in a situation in which group B has a higher mean than group A when examining the data but the posterior distributions show the credible values of group B being lower than that of group A.

So careful with strong priors and small, unequal sample sizes!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.