Importance of the prior

Question

The maximum a posteriori objective can be written as

$$\widehat{\theta}_\textrm{MAP} = \operatorname*{argmax}_\theta \log P(y\mid\theta) + \log P(\theta)$$

where $\log P(\theta)$––the prior––is a term that does not depend on the data.

I can see how the prior has regularization effects in the context of optimization, but I'm wondering what its importance is in Bayesian statistics. If priors are supposed to encode "a priori" information of parameters that might be useful during training, then why is it that they are often drawn from probability distributions?

For example, Goodfellow et al. suggests that priors should come from high-entropy distributions, such as the normal distribution, because this reflects uncertainty in the parameters. However, it seems counterintuitive to me to introduce a "random" prior if the point of the prior is to reduce uncertainty. How can these two claims be resolved?

Answer 1

The purpose of a prior is not to reduce uncertainty per se. Rather it is to express what is already known -- and generally agreed upon -- about the parameter or parameters in the model likelihood. Priors can be informative or uniformative. If little is known about a parameter, then an uniformative prior (maximally entropic) is appropriate. But often, we know more than we think we do. For instance, we know variance cannot be negative, so a prior that sets the prior probability to zero for $\sigma^2 < 0$ is sensible.

Pleading ignorance gets old quickly, and informative priors are more fun. If you are doing a series of experiments over several days, your prior today may be your posterior from yesterday. You can daisy-chain several days of results together this way.

For algebraic convenience, priors are usually chosen to be "conjugate" to the likelihood function, so the functional form of the posterior is the same as the prior. The best example of this is the beta function prior for the Bernoulli/binomial model likelihood.

Worth noting: if two researchers disagree on a prior, then they will disagree on the interpretation of the results of the experiment. That is not wrong; it is simply unfortunate, and the resolution is for them to agree on the prior, ideally a priori (before the fact), but a posteriori (after the fact) works, too.

As long as a prior does not insist that a particular value for a parameter has a probability identically equal to zero, then any prior will eventually shift toward the MLE, as enough data comes in. I think MLK Jr. was enunciating a basic Bayesian truth when he said "the arc of the moral universe is long, but it bends toward justice." Yes, priors/prejudices disappear when a) enough data is collected, and b) we keep our minds open and do not set any prior probabilities exactly to zero.

Answer 2

If priors are supposed to encode "a priori" information of parameters that might be useful during training, then why is it that they are often drawn from probability distributions?

The prior is a distribution or mass function in that the sum of all possible cases must sum to one. That the priors often look like well-known distributions is often a mathematical and computational convenience. It isn't mandatory as long as you sum to unity. There is also the case where you can use improper priors where the posterior is assured to sum to unity, but those are also computational conveniences.

I certainly have used non-standard distributions for the prior, that is, it is definitely not a named distribution. In fact, I have used piecewise distributions for continuous functions to reflect the known information. I can promise you, however, that doing that is computationally inconvenient. Encoding if or case statements in your prior isn't a joy if you are concerned about speed or using nice library functions.

priors should come from high-entropy distributions, such as the normal distribution because this reflects uncertainty in the parameters. However, it seems counterintuitive to me to introduce a "random" prior if the point of the prior is to reduce uncertainty. How can these two claims be resolved?

The argument for adding entropy to a prior is to reflect the idea that you may be wrong in your prior information. An example I often use involves magical quarters minted under the authority of the Congress where they are either double-sided, or biased one-third, one-half, or two-thirds for heads. It makes for all kinds of nice thought experiments.

Let us imagine you need to predict a coin from its tosses. You are not allowed to see it. You phone the US Mint and find out that they produce the coins in a ratio of 5% double headed, 20% 1/3rd bias for heads, 50% fair, 20% biased for heads, and 5% with double tails. That would be your prior, maybe.

Imagine you were going to go to a particular credit union to get a roll of quarters to run your test on. You were going to use this method until one of your researchers asks "how do you know they distribute coins proportionally to all institutions? What if some institutions only receive one type of coin?"

You realize that the researcher is correct. You do not want to discard this information from the Mint, but you do want to account for the fact that you may not have access to a representative sample. This gives you two choices.

The ignorance prior would be to grant all five choices equal weight. If you did that you would be ignoring real information. You could, however, multiply your informative distribution by the uniform distribution. The interesting consequence is that you recover your original informative distribution. It is true that your credit union may only receive double-headed coins, but the system still has to get proportionate amounts, and you do not know which luck of the draw distortion is going to your institution.

The second would be to note that you could reduce the weight of the center and increase the weight of the tails. Maybe the fair coin could be reduced to 40% and the tails increased to 10% each. The problem with this is that it isn't based on real information. There is no reason to widen the tails and shrink the center except a fear that a prior will have too strong of an effect on the outcome.

Now let us imagine you get eight coin tosses, and you have decided to use the MAP estimator as your decision function. Will a prior alter any of the answers? Yes.

For the informative prior the MAP estimator is:

Note at two and six successes how little of a difference between the MAP and first runner up. This is similar to the Frequentist concept of power. In those two cases, there isn't a lot separating two of the possible hypotheses.

Now consider the uniform distribution, the maximum entropy hypothesis. It results in a posterior and MAP of:

One interesting takeaway is that the most common real-world case is now rarely the answer under the MAP.

What this is telling you is that in four cases, the MAP is being chosen by the choice of the prior. Part of the issue is the small sample size, of course, but part is a lesson in the role an informative prior may have.

In five of nine cases, the data trumps the prior as they are distinct enough and unusual enough to provide sufficient information to the user. In four cases, there isn't enough information in the data to overcome the prior bias, which is that the fair coin cases are ten times more likely than either double-sided case and five times more likely than either biased case.

In a more real-world case, I am working on a class of time series whose parameters have certain theoretical properties that can be used to construct a prior. In simulations, I end up with relative efficiencies of around 20-1 gains over the Frequentist counter-part. I don't want a maximum entropy method because I do want to find the parameter, however, I do use a relatively agnostic prior over one of the nuisance parameters because I don't have a good reason to favor one location over another over a pretty wide range.

The prior should never be random, but it can be diffuse. It should reflect what you know.

You are still free to use your mixed distribution, which gives a third set of MAP estimators. They beg the question of why you chose that prior though.

Unfortunately, the best solution is to struggle with the prior, document it, and perform a sensitivity analysis.

This answer is also the Frequentist and Fisherian Likelihoodist argument against Bayesian methods. Is it a virtue to have prior knowledge if it is imperfect prior knowledge?