# How to choose prior in Bayesian parameter estimation

I know 3 methods to do parameter estimation, ML, MAP and Bayes approach. And for MAP and Bayes approach, we need to pick priors for parameters, right?

Say I have this model $p(x|\alpha,\beta)$, in which $\alpha,\beta$ are parameters, in order to do the estimation using MAP or Bayes, I read in the book that we'd better pick a conjugate prior $p(\alpha,\beta)$, which is a joint probability of $\alpha,\beta$, right?

I have 2 questions:

1. Do we have other choices picking the prior other than this conjugate one?

2. Can we pick priors for $\alpha$ and $\beta$ respectively like $p(\alpha)$ and $p(\beta)$, other than put them together in a joint one?

• Depending on what software you'll use, priors certainly don't have to be conjugate to the likelihood function... first and foremost, you should make sure that your priors represent your prior beliefs about the distribution of the parameters – Patrick Coulombe Dec 5 '13 at 2:02
• So I could just pick priors respectively for the parameters,right ? ACTUALLY I just try to understand baysian linear regression, no specific software considered – avocado Dec 5 '13 at 2:42
• Look up prior elicitation, e.g. here – Scortchi - Reinstate Monica Dec 5 '13 at 10:44

As stated in comment, the prior distribution represents prior beliefs about the distribution of the parameters.

When prior beliefs are actually available, you can:

• convert them in terms of moments (e.g. mean and variance) to fit a common distribution to these moments (e.g. Gaussian if your parameter lies to the real line, Gamma if it lies to $R^+$).
• use your intuitive understanding of these beliefs to propose a given prior distribution and check if it really fits your purpose and that it is not to sensitive to arbitrary choices (performing a robustness or sensibility analysis)

When no explicit prior beliefs are available, you can:

• derive (or simply use if already available, a great ressource is http://www.stats.org.uk/priors/noninformative/YangBerger1998.pdf) a Jeffreys (e.g. uniform for a location parameter) or a reference prior (especially in case of multivariate parameters).
• sometimes such choices are impossible or quite difficult to derive and in this case, you can try to choose among one of the many "generic" weakly informative prior (e.g. uniform shrinkage distribution for scale parameters of hierarchical model or $g$-prior for gaussian regression).

Having said that, there is no restriction to use a joint or an independent prior ($p(a,b)$ Vs $p(a) \cdot p(b)$). As a complement, I would say that in my humble opinion, there are three major things to take care when choosing a prior:

• take care that your posterior is integrable almost everywhere (or proper), which is always true if you use an integrable prior (see Does the Bayesian posterior need to be a proper distribution? for more details),
• limit the support of your prior only if you are highly confident on the support bounds (so avoid to do it).
• and last but not least, make sure (most of the time experimentally) that your choice of prior means what you want to express. To my opinion, this task is sometimes the more critical. Never forgot, that when doing inference a prior means nothing by itself, you have to consider the posterior (which is the combination of prior and likelihood).
• Thank you so much, could you please recommend me some tutorial stuff on how to do this kind of bayesian inference? – avocado Dec 5 '13 at 9:48
• @loganecolss You're welcome, I was personnaly a bit lost a few months ago and this post is simply the summary of my self-study and I am happy if it can help somebody else. Concerning your question, what do you mean by "this kind of bayesian inference" ? – peuhp Dec 5 '13 at 9:56
• I'm also self-studying machine learning stuff, I knew ML, but this bayesian approach of parameter estimation is new to me, hope you can show me some material to learn bayesian estimation and inference, ;-) – avocado Dec 5 '13 at 10:04
• @loganecolss, This is a good summary of the MLE, MAP and Bayesian inference. And this link give s a good summary of how to incorporate a prior to Bayesian inference for a binomial distribution. – Zhubarb Dec 5 '13 at 10:58
• A minor elaboration: A proper prior represents a consistent set of beliefs about the parameters. They don't have to be your beliefs. Indeed models are often more persuasive when they are some else's. – conjugateprior Mar 22 '16 at 20:11

There is also empirical Bayes. The idea is to tune the prior to the data:

$$\text{max}_{p(z)} \int p(\mathcal{D}|z)p(z) dz$$

While this might seem awkward at first, there are actually relations to minimum description length. This is also the typical way to estimate the kernel parameters of Gaussian processes.

To answer the two questions above directly:

1. You have other choices to choose non-conjugate priors other than conjugate priors. The problem is that if you choose non-conjugate priors, you cannot make exact Bayesian inference (simply put, you cannot derive a close-form posterior). Rather, you need to make approximate inference or use sampling methods such as Gibbs sampling, Rejection sampling, MCMC, etc. to derive you posterior. The problem with sampling methods is that intuitively, it's like drawing a picture of elephant in darkness by repetitively touching it----you may be biased and incomplete. The reason people choose non-conjugate prior is that for certain likelihood, conjugate prior option is pretty limited, or to say, most are non-conjugate.

2. Yes, you definitely can. If α and β are independent, which is the idealistic condition, you can derive their joint distribution by p(α)p(β). If they are not independent, you may need to figure out the conditional probability and do integral to derive joint distribution.