What happens if I choose a bad prior for Bayesian AB testing? I'm trying to go through this article http://developers.lyst.com/data/2014/05/10/bayesian-ab-testing/
and I see that they choose a Beta(3, 50) prior and make an argument for that. However, if you use their calculator and choose a Beta(1, 1) prior, the results don't seem too different. In fact I've been playing around with both small and large sample sizes, and a Beta(1, 1) prior often times generates a higher probability that A > B. What are some downfalls of using the wrong prior. Does our probability estimate mean less?
 A: You can't really say a prior is wrong, because it is up to you to decide what the prior is. One comfort is this: under not too restricting conditions your prior does not matter (if you have enough data). This is Bernstein Von Mises theorem.
A: You could have unexpected issues arise with strong priors, especially with low, unequal sample sizes.  For example, say, in an AB test, you specify priors for your conditions in which they both have the same mean and high precision (i.e., little variance in plausible parameter values).  If you have, say, lots of data for your group A (control group) and little data for your group B (test group), group A will be able to overcome the strong prior while group B may not be able to overcome the strong prior.  This could be an issue when say both the means of the data for group A and B are higher than the prior mean.  There will be enough data in group A for its posterior distribution to shift away from the prior but not enough for group B's to do so.  So you could end up in a situation in which group B has a higher mean than group A when examining the data but the posterior distributions show the credible values of group B being lower than that of group A.  
So careful with strong priors and small, unequal sample sizes! 
