How to choose prior in Bayesian parameter estimation

Question

I know 3 methods to do parameter estimation, ML, MAP and Bayes approach. And for MAP and Bayes approach, we need to pick priors for parameters, right?

Say I have this model $p(x|\alpha,\beta)$, in which $\alpha,\beta$ are parameters, in order to do the estimation using MAP or Bayes, I read in the book that we'd better pick a conjugate prior $p(\alpha,\beta)$, which is a joint probability of $\alpha,\beta$, right?

I have 2 questions:

1. Do we have other choices picking the prior other than this conjugate one?

2. Can we pick priors for $\alpha$ and $\beta$ respectively like $p(\alpha)$ and $p(\beta)$, other than put them together in a joint one?

Answer 1

As stated in comment, the prior distribution represents prior beliefs about the distribution of the parameters.

When prior beliefs are actually available, you can:

• convert them in terms of moments (e.g. mean and variance) to fit a common distribution to these moments (e.g. Gaussian if your parameter lies to the real line, Gamma if it lies to $R^+$).
• use your intuitive understanding of these beliefs to propose a given prior distribution and check if it really fits your purpose and that it is not to sensitive to arbitrary choices (performing a robustness or sensibility analysis)

When no explicit prior beliefs are available, you can:

• derive (or simply use if already available, a great ressource is http://www.stats.org.uk/priors/noninformative/YangBerger1998.pdf) a Jeffreys (e.g. uniform for a location parameter) or a reference prior (especially in case of multivariate parameters).
• sometimes such choices are impossible or quite difficult to derive and in this case, you can try to choose among one of the many "generic" weakly informative prior (e.g. uniform shrinkage distribution for scale parameters of hierarchical model or $g$-prior for gaussian regression).

Having said that, there is no restriction to use a joint or an independent prior ($p(a,b)$ Vs $p(a) \cdot p(b)$). As a complement, I would say that in my humble opinion, there are three major things to take care when choosing a prior:

• take care that your posterior is integrable almost everywhere (or proper), which is always true if you use an integrable prior (see Does the Bayesian posterior need to be a proper distribution? for more details),
• limit the support of your prior only if you are highly confident on the support bounds (so avoid to do it).
• and last but not least, make sure (most of the time experimentally) that your choice of prior means what you want to express. To my opinion, this task is sometimes the more critical. Never forgot, that when doing inference a prior means nothing by itself, you have to consider the posterior (which is the combination of prior and likelihood).

Answer 2

There is also empirical Bayes. The idea is to tune the prior to the data:

$$\text{max}_{p(z)} \int p(\mathcal{D}|z)p(z) dz$$

While this might seem awkward at first, there are actually relations to minimum description length. This is also the typical way to estimate the kernel parameters of Gaussian processes.

Answer 3

To answer the two questions above directly:

1. You have other choices to choose non-conjugate priors other than conjugate priors. The problem is that if you choose non-conjugate priors, you cannot make exact Bayesian inference (simply put, you cannot derive a close-form posterior). Rather, you need to make approximate inference or use sampling methods such as Gibbs sampling, Rejection sampling, MCMC, etc. to derive you posterior. The problem with sampling methods is that intuitively, it's like drawing a picture of elephant in darkness by repetitively touching it----you may be biased and incomplete. The reason people choose non-conjugate prior is that for certain likelihood, conjugate prior option is pretty limited, or to say, most are non-conjugate.

2. Yes, you definitely can. If α and β are independent, which is the idealistic condition, you can derive their joint distribution by p(α)p(β). If they are not independent, you may need to figure out the conditional probability and do integral to derive joint distribution.