I know 3 methods to do parameter estimation, ML, MAP and Bayes approach. And for MAP and Bayes approach, we need to pick priors for parameters, right?

Say I have this model $p(x|\alpha,\beta)$, in which $\alpha,\beta$ are parameters, in order to do the estimation using MAP or Bayes, I read in the book that we'd better pick a conjugate prior $p(\alpha,\beta)$, which is a joint probability of $\alpha,\beta$, right?

I have 2 questions:

  1. Do we have other choices picking the prior other than this conjugate one?

  2. Can we pick priors for $\alpha$ and $\beta$ respectively like $p(\alpha)$ and $p(\beta)$, other than put them together in a joint one?

  • 1
    $\begingroup$ Depending on what software you'll use, priors certainly don't have to be conjugate to the likelihood function... first and foremost, you should make sure that your priors represent your prior beliefs about the distribution of the parameters $\endgroup$ – Patrick Coulombe Dec 5 '13 at 2:02
  • $\begingroup$ So I could just pick priors respectively for the parameters,right ? ACTUALLY I just try to understand baysian linear regression, no specific software considered $\endgroup$ – avocado Dec 5 '13 at 2:42
  • 1
    $\begingroup$ Look up prior elicitation, e.g. here $\endgroup$ – Scortchi Dec 5 '13 at 10:44

As stated in comment, the prior distribution represents prior beliefs about the distribution of the parameters.

When prior beliefs are actually available, you can:

  • convert them in terms of moments (e.g. mean and variance) to fit a common distribution to these moments (e.g. Gaussian if your parameter lies to the real line, Gamma if it lies to $R^+$).
  • use your intuitive understanding of these beliefs to propose a given prior distribution and check if it really fits your purpose and that it is not to sensitive to arbitrary choices (performing a robustness or sensibility analysis)

When no explicit prior beliefs are available, you can:

  • derive (or simply use if already available, a great ressource is http://www.stats.org.uk/priors/noninformative/YangBerger1998.pdf) a Jeffreys (e.g. uniform for a location parameter) or a reference prior (especially in case of multivariate parameters).
  • sometimes such choices are impossible or quite difficult to derive and in this case, you can try to choose among one of the many "generic" weakly informative prior (e.g. uniform shrinkage distribution for scale parameters of hierarchical model or $g$-prior for gaussian regression).

Having said that, there is no restriction to use a joint or an independent prior ($p(a,b)$ Vs $p(a) \cdot p(b)$). As a complement, I would say that in my humble opinion, there are three major things to take care when choosing a prior:

  • take care that your posterior is integrable almost everywhere (or proper), which is always true if you use an integrable prior (see Does the Bayesian posterior need to be a proper distribution? for more details),
  • limit the support of your prior only if you are highly confident on the support bounds (so avoid to do it).
  • and last but not least, make sure (most of the time experimentally) that your choice of prior means what you want to express. To my opinion, this task is sometimes the more critical. Never forgot, that when doing inference a prior means nothing by itself, you have to consider the posterior (which is the combination of prior and likelihood).
  • $\begingroup$ Thank you so much, could you please recommend me some tutorial stuff on how to do this kind of bayesian inference? $\endgroup$ – avocado Dec 5 '13 at 9:48
  • $\begingroup$ @loganecolss You're welcome, I was personnaly a bit lost a few months ago and this post is simply the summary of my self-study and I am happy if it can help somebody else. Concerning your question, what do you mean by "this kind of bayesian inference" ? $\endgroup$ – peuhp Dec 5 '13 at 9:56
  • 1
    $\begingroup$ I'm also self-studying machine learning stuff, I knew ML, but this bayesian approach of parameter estimation is new to me, hope you can show me some material to learn bayesian estimation and inference, ;-) $\endgroup$ – avocado Dec 5 '13 at 10:04
  • 1
    $\begingroup$ @loganecolss, This is a good summary of the MLE, MAP and Bayesian inference. And this link give s a good summary of how to incorporate a prior to Bayesian inference for a binomial distribution. $\endgroup$ – Zhubarb Dec 5 '13 at 10:58
  • $\begingroup$ A minor elaboration: A proper prior represents a consistent set of beliefs about the parameters. They don't have to be your beliefs. Indeed models are often more persuasive when they are some else's. $\endgroup$ – conjugateprior Mar 22 '16 at 20:11

There is also empirical Bayes. The idea is to tune the prior to the data:

$$\text{max}_{p(z)} \int p(\mathcal{D}|z)p(z) dz$$

While this might seem awkward at first, there are actually relations to minimum description length. This is also the typical way to estimate the kernel parameters of Gaussian processes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.