17
$\begingroup$

I am currently reading about Bayesian Methods in Computation Molecular Evolution by Yang. In section 5.2 it talks about priors, and specifically Non-informative/flat/vague/diffuse, conjugate, and hyper- priors.

This might be asking for an oversimplification but, could someone explain simply the difference between these types of priors and how that affects the outcome of an analysis/decisions I would make during the process of a Bayesian analysis?

(I'm not a statistician and I am just starting out on the road to learning Bayesian analyses so the more it is in layman terms the better)

$\endgroup$

2 Answers 2

17
$\begingroup$

Simply put, a flat/non-informative prior is used when one has little/no knowledge about the data and hence it has the least effect on outcomes of your analysis (i.e. posterior inference).

Conjugate distributions are those whose prior and posterior distributions are the same, and the prior is called the conjugate prior. It is favoured for its algebraic conveniences, especially when the likelihood has a distribution in the form of exponential family (Gaussian, Beta, etc.). This is hugely beneficial when carrying posterior simulations using Gibbs sampling.

And finally imagine that a prior distribution is set on a parameter in your model, however you want to add an another level of complexity/uncertainty. You would then impose a prior distribution on the parameters of the aforementioned prior, hence the name hyper-prior.

I think Gelman's Bayesian Data Analysis is a great start for anyone who's interested in learning Bayesian statistics:)

$\endgroup$
1
  • 2
    $\begingroup$ Second the endorsement of Bayesian Data Analysis. $\endgroup$
    – Sycorax
    Commented Jul 30, 2013 at 17:40
16
$\begingroup$

At the highest level, we can think of all manner of priors as specifying some amount of information that the researcher brings to bear on the analysis outside of the data itself: before looking at the data, which values of parameters are more likely?

In the dark ages of Bayesian analysis, when the Bayesians were fighting it out with frequentists, there was a belief that the researcher would want to introduce as little information to the analysis via the prior as possible. So there was a lot of research and argument devoted to understanding how, precisely, a prior could be "non-informative" in this way. Today, Gelman argues against the automatic choice of non-informative priors, saying in Bayesian Data Analysis that the description "non-informative" reflects his attitude towards the prior, rather than any "special" mathematical features of the prior. (Moreover, there was a question in the early literature of at what scale a prior is noninformative. I don't think that this is especially important to your question, but for a good example of this argument from a frequentist perspective, see the beginning of Gary King, Unifying Political Methodology.)

A "flat" prior indicates a uniform prior where all values in the range are equally likely. Again, there are arguments to be had about whether these are truly non-informative, since specifying that all values are equally likely is, in some way, information, and may be sensitive to how the model is parameterized. Flat priors have a long history in Bayesian analysis, stretching back to Bayes and Laplace.

A "vague" prior is highly diffuse though not necessarily flat, and it expresses that a large range of values are plausible, rather than concentrating the probability mass around specific range. Essentially, it is a prior with high variance (whatever "high" variance means in your context).

Conjugate priors have the convenient feature that, when multiplied by the appropriate likelihood, they produce a posterior distribution with a “nice” expression. (There is some nuance here; see Do conjugate priors just lead to a posterior that is a modification of the parameters of the prior?) One example of this is the beta prior with the binomial likelihood, or the gamma prior with the poisson likelihood. There are helpful tables of these all over the Internet and Wikipedia. The exponential family is extremely convenient in this regard.

Conjugate priors are often the "default" choice for some problems because of their convenient properties, but this does not necessarily mean that they are the "best" unless one's prior knowledge can be expressed via the conjugate prior. Advances in computation mean that conjugacy is not as prized as it once was (cf Gibbs sampling vs NUTS), so we can more easily perform inference with non-conjugate priors without much trouble.

Hyper-priors are priors on the prior. This means that rather than specifying, say, a $N(\mu,\sigma^2)$ prior on a parameter with fixed $\mu$ and $\sigma^2$, you might express a prior on the parameter $\mu$ and a prior on the parameter $\sigma^2$. Most often, this is used in hierarchical modeling, when you believe that there is a common feature to all of the data points in question (say, because you are performing a statistical analysis on replications of the same experiment), and that variation in the data is explained as being caused by random assignment of parameters from this common distribution to the data points.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.