How do Bayesian Statistics handle the absence of priors?

Question

This question was inspired by two recent interactions I had, one here in CV, the other over at economics.se.

There, I had posted an answer to the well-known "Envelope Paradox" (mind you, not as the "correct answer" but as the answer flowing from specific assumptions about the structure of the situation). After a time a user posted a critical comment, and I engaged in conversation trying to understand his point. It was obvious that he was thinking the Bayesian way, and kept talking about priors -and then it dawned on me, and I said to my self:"Wait a minute, who said anything about any prior? In the way I have formulated the problem, there are no priors here, they just don't enter the picture, and don't need to".

Recently, I saw this answer here in CV, about the meaning of Statistical Independence. I commented to the author that his sentence

"... if events are statistically independent then (by definition) we cannot learn about one from observing the other."

was blatantly wrong. In a comment exchange, he kept returning to the issue of (his words)

"Wouldn't "learning" mean changing our beliefs about a thing based on observation of another? If so, doesn't independence (definitionally) preclude this?

Once again, it was obvious that he was thinking the Bayesian way, and that he considered self-evident that we start by some beliefs (i.e. a prior), and then the issue is how we can change/update them. But how the first-first belief is created?

Since science must conform to reality, I note that situations exist were the human beings involved have no priors (I, for one thing, walk into situations without any prior all the time -and please don't argue that I do have priors but I just don't realize it, let's spare ourselves bogus psychoanalysis here).

Since I happened to have heard the term "uninformative priors", I break my question in two parts, and I am pretty certain that users here that are savvy in Bayesian theory, know exactly what I am about to ask:

Q1: Is the absence of a prior equivalent (in the strict theoretical sense) to having an uninformative prior?

If the answer to Q1 is "Yes" (with some elaboration please), then it means that the Bayesian approach is applicable universally and from the beginning, since in any instance the human being involved declares "I have no priors" we can supplement in its place a prior that is uninformative for the case at hand.

But if the answer to Q1 is "No", then Q2 comes along:

Q2 : If the answer to Q1 is "No", does this mean that, in cases where there are no priors, the Bayesian approach is not applicable from the beginning, and we have to first form a prior by some non-Bayesian way, so that we can subsequently apply the Bayesian approach?

Answer 1

Q1: Is the absence of a prior equivalent (in the strict theoretical sense) to having an uninformative prior?

No.

First, there is no mathematical definition for an "uninformative prior". This word is only used informally to describe some priors.

For example, Jeffrey's prior is often called "uninformative". This prior generalizes the uniform prior for translation invariant problems. Jeffrey's prior somehow adapts to the (information theoretic) Riemannian geometry of the model and thus is independent of parametrization, only dependent on the geometry of the manifold (in the space of distributions) that is the model. It might be perceived as canonical, but it's only a choice. It's just the uniform prior according to Riemannian structure. It's not absurd to define "uninformative = uniform" as a simplification of the question. This applies to many cases and helps to ask a clear and simple question.

Doing Bayesian inference without a prior is like "how can I guess $E(X)$ without any assumption about the distribution of $X$ only knowing that $X$ has values in $[0;1]$?" This question obviously makes no sense. If you answer 0.5, you probably have a distribution in mind.

The Bayesian and frequentist approaches simply answer different questions. For example, about estimators which is maybe the simplest:

• Frequentist (for example): "How can I estimate $\theta$ such that my answer has the smallest error (only averaged over $x$) in the worst case (over $\theta$)?". This leads to minimax estimators.

• Bayesian: "How can I estimate $\theta$ such that my answer has the smallest error in average (over $\theta$) ?". This leads to Bayes estimators. But the question is incomplete and must specify "average in what sense?". Thus the question is only complete when it contains a prior.

Somehow, frequentist aims at worst case control and does not need a prior. Bayesian aims at average control and requires a prior to say "average in what sense?".

Q2 : If the answer to Q1 is "No", does this mean that, in cases where there are no priors, the Bayesian approach is not applicable from the beginning, and we have to first form a prior by some non-Bayesian way, so that we can subsequently apply the Bayesian approach?

Yes.

But beware of canonical prior construction. It might sound mathematically appealing but is not automatically realistic from a Bayesian point of view. It is possible a mathematically nice prior actually corresponds to a dumb belief system. For example if you study $X\sim N(\mu,1)$, Jeffrey's prior on $\mu$ is uniform and if about people's average size, this might not be a very realistic system. However with only a few observations, the problem actually disappears quite fast. The choice is not very important.

True problems with prior specification happen in more complicated problems in my opinion. What is important here is to understand what a certain prior says.

Answer 2

First of all, Bayesian approach is often used because you want to include prior knowledge in your model to enrich it. If you don't have any prior knowledge, then you stick to so-called "uninformative" or weekly informative priors. Notice that uniform prior is not "uninformative" by definition, since assumption about uniformity is an assumption. There is no such a thing as a truly uninformative prior. There are cases where "it could be anything" is a reasonable "uninformative" prior assumption, but there are also cases where stating that "all values are equally likely" is a very strong and unreasonable assumption. For example, if you assumed that my height can be anything between 0 centimeters and 3 meters, with all of the values being equally likely a priori, this wouldn't be a reasonable assumption and it would give too much weight to the extreme values, so it could possibly distort your posterior.

On another hand, Bayesian would argue that there is really no situations where you have no prior knowledge or beliefs whatsoever. You always can assume something and as a human being, you're doing it all the time (psychologists and behavioral economists made tons of research on this topic). The whole Bayesian fuss with the priors is about quantifying those preconception and stating them explicitly in your model, since Bayesian inference is about updating your beliefs.

It is easy to come up with "no prior assumptions" arguments, or uniform priors, for abstract problems, but for real-life problems you'd have prior knowledge. If you needed to make a bet about amount of money in an envelope, you'd know that the amount needs to be non-negative and finite. You also could make an educated guess about the upper bound for the possible amount of the money given your knowledge about the rules of the contest, funds available for your adversary, knowledge about physical size of the envelope and the amount of money that could physically fit in it, etc. You could also make some guesses about the amount of money that your adversary could be willing to put in the envelope and possibly loose. There is lots of things that you would know as a base for your prior.

Answer 3

This is only a short remark as addition to the other excellent answers. Often, or at least sometimes, it is somewhat arbitrary (or conventional) what part of the information entering a statistical analysis is called data and which part is called prior. Or, more generally, we can say that information in a statistical analysis comes from three sources: the model, the data, and the prior. In some cases, such as linear models or glm's, the separation is quite clear, at least conventionally.

I will reuse an example from Maximum Likelihood Estimation (MLE) in layman terms to illustrate my point. Say a patient enters a physician's office, with some medical problems that turn out to be difficult to diagnose. This physician hasn't seen something quite similar before. Then, talking with the patient it surfaces some new information: this patient visited tropical Africa quite recently. Then it appears to the physician that this could be malaria or some other tropical disease. But note, that this information is clearly to us data, but at least in many statistical models that could be used, it will enter the analysis in the form of a prior distribution, a prior distribution giving higher probability to some tropical diseases. But we could, maybe, make some (larger), more complete model, where this information enters as data. So, at least in part, the distinction data/prior is conventional.

We are used to, and accept, this convention because of our emphasis on some classes of conventional models. But, in the larger scheme of things, outside the world of stylized statistical models, the situation is less clear.

Answer 4

question 1 I think the answer is probably no. My reason is we don't really have a definition for "uninformative" except for somehow measuring how far the final answer is from some arbitrarily informative model/likelihood. Many uninformative priors are validated against "intuitive" examples where we already have "the model/likelihood" and "the answer" in mind. We then ask the uninformative prior to give us the answer we want.

My problem with this is I struggle with believing that someone can have a really good, well informed model or model structure for their population, and simultaneously have "no information" about likely and unlikely parameter values for that model. For example using logistic regression, see "A WEAKLY INFORMATIVE DEFAULT PRIOR DISTRIBUTION. FOR LOGISTIC AND OTHER REGRESSION MODELS"

I think the discrete uniform prior is the only one we could reasonably say is the "first-first" prior. But you run into problems of using it, thinking you have "no information", but then suddenly having reactions to "unintuitive" answers (hint: if you don't like a bayesian answer - you might have left information out of the prior or likelihood!). Another problem you run into is getting the discretisation right for your problem. And even thinking of this, you need to know the number of discrete values to apply the discrete uniform prior.

Another property to consider for your prior is the "tail behaviour" relative to the likelihood you are using.

on to question 2

Conceptually, I don't see anything wrong with specifying a distribution without the use of a prior or likelihood. You can start a problem by saying "my pdf is ... and I want to calculate ... wrt this pdf". Then you are creating a constraint for the prior, prior predictive, and likelihood. The bayesian method is for when you have a prior and a likelihood, and you want to combine them into a posterior distribution.

It's probably a matter of being clear on what your probabilities are. Then the argument shifts to "does this pdf/pmf represent what I say it represents?" - which is the space you want to be in I think. From your example, you are saying the single distribution reflects all the available information - there is no "prior" because it's already contained (implicitly) in the distribution you are using.

You can also apply bayes in reverse - what "prior", "likelihood" and "data" gives me the actual prior I am considering? This is one way you can see that a $U (0,1) $ prior for a $Bin(n,p) $ likelihood "looks" like it corresponds to a "posterior" for a $Beta (0,0) $ "prior" with $2$ observations - $1$ from each category.

on the so called blatantly wrong comment

To be honest, I would be very interested to see how any numbet of observation could be used to predict a "statistically independent" observation. As an example, if I tell you I'll generate 100 standard normal variables. I give you 99, and get you to give me your best prediction for 100th. I say you cannot make a better prediction for the 100th than 0. But this is the same you would predict for the 100th if I gave you no data. Hence you learn nothing from the 99 data points.

However, if I tell you that it was "some normal distribution", you can use the 99 data points to estimate the parameters. Then the data are now no longer "statistically independent", because we learn more about the common structure as we observe more data. Your best prediction now uses all 99 data points