Intuition about the deep meaning of Bayesian priors and its influence on posteriors

Question

In estimating posterior distributions, Bayesians rely on the idea of the prior distribution. In many examples, I see this being set fairly arbitrarily, ie ~N(0,1).

It's clear that the posterior is integrally related to the prior, however, the fact that it is arbitrary seems unsatisfactory. Furthermore, the prior affects the posterior in a qualitative way. Viewed in the sense of a "weighted" average, this is further unsatisfactory.

My two specific questions are:

What is the value of having priors when they seem arbitrary, or are handwaved or calculated away at a later stage? If this dismissive description of attention of priors is correct, and since priors are so integral to Bayesian stats, doesn't this undermine the practice or theory of Bayesian stats?

Related question: don't priors have considerable influence on the resulting posterior? (If they do not, the question above applies). If they do, and since apparently not much thought is given to them, doesn't this just push the "philosophy" and "content" of Bayesian stats into a black box?

I freely admit this question doesn't seem right (or is embarrassing to me). Where has my education about Bayesian stats gone wrong?

Answer 1

Your statement echoes Jaynes. He said

When we look at these problems on a sufficiently fundamental level and realize how careful one must be to specify the prior information before we have a well-posed problem, it becomes evident that there is, in fact, no logical difference between (3.51) and (4.3); exactly the same principles are needed to assign either sampling probabilities or prior probabilities, and one man’s sampling probability is another man’s prior probability.

The equations in chapter three are on elementary sampling theory and four on elementary hypothesis testing.

There are three primary methods to create Bayesian theory from Cox's, de Finetti's and Savage's axioms. Cox is built on logic, de Finetti on gambling and Savage on preferences.

In all three cases, you do not get well-posed problems with arbitrary calculations. If you think of a probability statement as a statement about a logical assertion, then to get a proper answer all parts of the logical argument must be included. Likewise, when one gambles, one would be insane to purposely ignore information about which you were going to gamble on. Finally, it begs rationality with respect to preferences to ignore knowledge.

I believe the mistake comes from a misunderstanding of probabilities as frequencies. They are not. Long run frequencies will not be derived from Bayesian methods. It can be the case they have nice Frequentist properties, but this is incidental.

Now, this does ask is there any circumstance where one should ignore prior information and the answer is "yes." As long as one is not introducing contradictory information or damaging the assertion there can be. Consider the case of a high dimension model with prohibitive calculation costs that could be approximated closely by an approximate solution. Weakening the prior gives a solution when a strong prior makes it impossible to do the work. Likewise, consider a low dimension model where time is of the essence and determining a full prior would result in catastrophic losses due to the time constraints. This is the terrorist with the bomb scenario. In that case, it is rational to use less than the available information.

Laziness or ignorance is not an excuse for ignoring the prior.

Answer 2

This is how I read your question -- "Why are priors given arbitrary values when they have a bearing on the calculated posterior?"

Note: I come from a physics background -- please let me know if you think I am using some terms wrong.

I shall pose a series of atomic questions and answer them as I understand them from the perspective of Bayesian statistics.

1. Notation and terms: I think of a system connected to each other causally in pairs (think of a directed graph). The quantities are divided into Query, Hidden and Evidence. The posterior probability is given by $P(Q|E) = \sum_H P(Q|E,H)P(H)$, it is marginal with respect to the variables in the Hidden($H$) class.
   The statement of Bayes' Theorem is
   $\underbrace{P(Q|E)}_{\text{posterior distribution of Q given E}} = \frac{{\overbrace{P(E|Q)}^{\text{likelihood of E given Q}}}\;\times\;{\overbrace{P(Q)}^{\text{prior distribution of Q}}}}{\sum_{Q'}P(E|Q')P(Q')}$
2. Is the prior important to the calculation of the posterior?: Given enough evidence/data and a simple enough event space, no. But a suitable choice of the prior can lead to the "correct" posterior given less evidence/data or less iterations.
3. Is the prior distribution ignored in practice?: Given enough data, it is not important because you can use the valuable time you have to other things. But having a prior distribution available from experiments enables better sanity checks (tests or debugging) once the posterior is obtained.
4. When is the prior distribution important at all?
   1. Less data is available
   2. Multiple similar competing hypotheses (correlated with having a larger event space)
   3. Philosophically important to Jayne's (or what I subjectively believe is Jayne's explanation of the prior -- I haven't had a lot of time to assimilate it yet) approach.
5. What makes sense for statistical mechanics? It is fine to seek explanations over discrete event spaces without referring to the priors. But faced with multidimensional systems on which most problems scale as the factorial, it seems to me that maximizing the entropy over the given constraints IS a very pragmatic way. But then I,as a beginner, haven't had enough time to understand if this is the only/best possible choice.

Answer 3

My possibly idiosyncratic view is as follows. If we had an exact, fully-known, prior distribution on the parameters, possibly belief-based, and we knew the true likelihood function, the Bayesian paradigm gives us the optimal way of updating that prior with the likelihood to get a posterior. In real life we don't have either a prior or a likelihood, except in what seems to me to be rare cases, so we apply an intuitive "smoothness in function space" argument that runs as follows. As long as the prior we use for calculation is close to the real, unobservable prior we have, and the likelihood function we use for calculation is close to the real, usually unknowable likelihood function, applying the Bayesian paradigm will get us a posterior that is close to the real, uncalculatable posterior. Applying the Bayesian paradigm even with approximate priors and likelihoods is likely to get us closer (on average) than doing something else, because it eliminates a source of noise in the move from prior to posterior - noise due to using a suboptimal updating algorithm.

This, then, is the value of trying to state your prior information as a probability distribution - it allows you to use the optimal updating algorithm, thereby reducing the error in the beliefs you form after you've looked at the data.

As a rather lengthy side note, this implies that Bayesian robustness is a desirable feature of our overall process (assigning priors and likelihood functions, performing the update calculations), the more so as our confidence in the accuracy of our constructed / assumed prior and likelihood functions degrades. At some point, we'll have so little confidence in our ability to form any sort of reasonable approximation to one, the other, or both, that we may as well abandon the Bayesian paradigm and do something else. Alternatively, the cost of setting up and executing the Bayesian paradigm may be so great, relative to the gains thereof, that we are, again, better off doing something else, such as running a classical t-test, observing a t-statistic of 19.4, and rejecting the null hypothesis that we created just to make life simpler.

Now, as to the influence priors have - that depends on the prior, the likelihood function, and the data. It is quite easy to find all sorts of real-world situations where the data overwhelms the prior, in which case even very different priors lead to very similar posteriors. In these situations, worrying about the likelihood is far more important than worrying about the prior. On the other hand, in situations where getting data is very cost or time-intensive, the prior information may have to be carefully extracted from the relevant experts in order to make the best use of it as we can. (This was the case in my previous job, in which I did reliability analysis for solar panels and trackers, among other things - testing a large, expensive piece of equipment that is supposed to track the sun to derive a mean time to failure is both time-consuming and expensive.) So, the influence of the prior is situational, and that same situationality drives where we should focus our efforts in order to make best use of the optimal updating algorithm that Bayes' Theorem gives us.