I am trying to get a grasp of the case that a very intuitive initial prior provided to a Bayesian Inference process turns out to be entirely wrong. Is there a standard model for the effect of error in an initial prior?

That might just get me more comfortable in picking an initial prior and choosing Bayesian Inference in the first place.

  • 1
    $\begingroup$ Not really, unless you can embed your prior within a family of priors and run a hierarchical Bayes inference over that family. There are however some Bayesian approaches to prior misspecification as posterior predictive and Evans' relative belief. $\endgroup$
    – Xi'an
    Mar 5 '16 at 20:42
  • $\begingroup$ Frankly I have found web.stanford.edu/class/msande226/lecture16_inference.pdf equally as helpful in providing a good perspective as any of the answers here. $\endgroup$
    – matt
    Mar 17 '16 at 20:03

What do you mean that the initial prior is "entirely wrong"? The prior is supposed to encode whatever information you have prior to seeing the data. It's only "wrong" if it fails to do that. A common way in which that can happen is overconfidence -- creating a prior that is unjustifiably narrow given the information you have.

Another common cause of unreasonable priors is because you lack an intuitive handle on what your model parameters mean. A strategy for dealing with that is to reparameterize in term of quantities that are more meaningful to your intuition. For example, a beta distribution is usually parameterized in terms of parameters $\alpha$ and $\beta$ that have no clear intuitive interpretation; reparameterizing in terms of the mean and standard deviation of the distribution makes it much easier to specify a reasonable prior.

If you still find a large mismatch between prior and likelihood, consider this a sign that something you thought was true isn't. Maybe there's a problem with your data collection. Maybe the process generating the data is different from what you thought, and you need to revise your model (not just revise parameter estimates). Or maybe your prior information was faulty, in which case Bayes' Rule creates a posterior distribution that reflects the appropriate compromise between your prior information and the new data.

  • $\begingroup$ Actually, all conjugate priors have an interpretation as hypothetical historical data (e.g. for a beta as the prior for a binomial the interpretation is having observed historical prior information equivalent to $y=\alpha$ out of $n=\alpha+\beta$ successes). $\endgroup$
    – Björn
    Mar 6 '16 at 6:40

One way of looking at it is to use mixture priors with the main components reflecting the best available prior information and one extra component being a very weakly informative component. If there is no-prior data conflict, then the informative mixture components will dominate the inference and if there is a prior-data conflict the non-informative component will dominate the inference. The interesting bit is that you can also look at the posterior weights of each mixture component. This approach is e.g. described by Schmidli, H., Gsteiger, S., Roychoudhury, S., O'Hagan, A., Spiegelhalter, D., & Neuenschwander, B. (2014). Robust meta‐analytic‐predictive priors in clinical trials with historical control information. Biometrics, 70(4), 1023-1032.. There are a few other approaches that do achieve similar things. As you can see in the references the key point for getting a robust analysis is to have long-tails in the prior, which typical conjugate distributions do not have (there is even some papers that say that they give the prior the maximum possible weight in inference - for a mixture of conjugate priors this is no longer the case in the same way, but you can still do simple conjugate updating).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.