# How do Bayesian Statistics handle the absence of priors?

This question was inspired by two recent interactions I had, one here in CV, the other over at economics.se.

There, I had posted an answer to the well-known "Envelope Paradox" (mind you, not as the "correct answer" but as the answer flowing from specific assumptions about the structure of the situation). After a time a user posted a critical comment, and I engaged in conversation trying to understand his point. It was obvious that he was thinking the Bayesian way, and kept talking about priors -and then it dawned on me, and I said to my self:"Wait a minute, who said anything about any prior? In the way I have formulated the problem, there are no priors here, they just don't enter the picture, and don't need to".

Recently, I saw this answer here in CV, about the meaning of Statistical Independence. I commented to the author that his sentence

"... if events are statistically independent then (by definition) we cannot learn about one from observing the other."

was blatantly wrong. In a comment exchange, he kept returning to the issue of (his words)

"Wouldn't "learning" mean changing our beliefs about a thing based on observation of another? If so, doesn't independence (definitionally) preclude this?

Once again, it was obvious that he was thinking the Bayesian way, and that he considered self-evident that we start by some beliefs (i.e. a prior), and then the issue is how we can change/update them. But how the first-first belief is created?

Since science must conform to reality, I note that situations exist were the human beings involved have no priors (I, for one thing, walk into situations without any prior all the time -and please don't argue that I do have priors but I just don't realize it, let's spare ourselves bogus psychoanalysis here).

Since I happened to have heard the term "uninformative priors", I break my question in two parts, and I am pretty certain that users here that are savvy in Bayesian theory, know exactly what I am about to ask:

Q1: Is the absence of a prior equivalent (in the strict theoretical sense) to having an uninformative prior?

If the answer to Q1 is "Yes" (with some elaboration please), then it means that the Bayesian approach is applicable universally and from the beginning, since in any instance the human being involved declares "I have no priors" we can supplement in its place a prior that is uninformative for the case at hand.

But if the answer to Q1 is "No", then Q2 comes along:

Q2 : If the answer to Q1 is "No", does this mean that, in cases where there are no priors, the Bayesian approach is not applicable from the beginning, and we have to first form a prior by some non-Bayesian way, so that we can subsequently apply the Bayesian approach?

• I will note that as a scientist "no prior belief" is a pretty extremist statement... akin to seeing only the static "snow" of an old analog TV set tuned to a dead station, and hearing only white noise. Manifestly scientists do not believe that nothing in the world relates to or carries information about anything... if we did believe that we would not be scientists. Of course, the Bayesian articulation of "uninformative" carries very general beliefs about possibility and probability. – Alexis Feb 2 at 18:07
• @Alexis Depends on the situation. For example in the "Envelop Paradox" the issue is, after looking at the amount the one envelop contains,whether or not I had any prior belief about whether what am I looking at is the "large" amount or the "small" amount. And I don't see anything extreme in declaring here "I have no prior belief on the matter". – Alecos Papadopoulos Feb 2 at 18:08
• Do you have a prior belief that there are some amounts and that they have some distribution between envelopes? (Even if you are agnostic as to the specific distribution or its parameterization?) – Alexis Feb 2 at 18:10
• @Alexis Sure, but it is formulated as structural knowledge that exists independently of subsequent information. It is not a belief that needs updating. And since this formulation reflects the perception of at least one member of the human race (myself), it is a real-world situation, and the question is whether it is considered amenable to Bayesian analysis or not. Of course for some other person that declares "I have the following prior belief about the distribution of amounts between envelopes", the applicability of the Bayesian approach is obvious. – Alecos Papadopoulos Feb 2 at 18:20
• You seem to believe that either there is NO prior or there is THE prior. In a Bayesian modelling as I see it, the posterior analysis is conditional or relative to the choice of A prior and I do not use this prior as absolute. My prior is a natural way of introducing a probability structure and measure on the parameter space. – Xi'an Feb 4 at 20:19

Q1: Is the absence of a prior equivalent (in the strict theoretical sense) to having an uninformative prior?

No.

First, there is no mathematical definition for an "uninformative prior". This word is only used informally to describe some priors.

For example, Jeffrey's prior is often called "uninformative". This prior generalizes the uniform prior for translation invariant problems. Jeffrey's prior somehow adapts to the (information theoretic) Riemannian geometry of the model and thus is independent of parametrization, only dependent on the geometry of the manifold (in the space of distributions) that is the model. It might be perceived as canonical, but it's only a choice. It's just the uniform prior according to Riemannian structure. It's not absurd to define "uninformative = uniform" as a simplification of the question. This applies to many cases and helps to ask a clear and simple question.

Doing Bayesian inference without a prior is like "how can I guess $E(X)$ without any assumption about the distribution of $X$ only knowing that $X$ has values in $[0;1]$?" This question obviously makes no sense. If you answer 0.5, you probably have a distribution in mind.

The Bayesian and frequentist approaches simply answer different questions. For example, about estimators which is maybe the simplest:

• Frequentist (for example): "How can I estimate $\theta$ such that my answer has the smallest error (only averaged over $x$) in the worst case (over $\theta$)?". This leads to minimax estimators.

• Bayesian: "How can I estimate $\theta$ such that my answer has the smallest error in average (over $\theta$) ?". This leads to Bayes estimators. But the question is incomplete and must specify "average in what sense?". Thus the question is only complete when it contains a prior.

Somehow, frequentist aims at worst case control and does not need a prior. Bayesian aims at average control and requires a prior to say "average in what sense?".

Q2 : If the answer to Q1 is "No", does this mean that, in cases where there are no priors, the Bayesian approach is not applicable from the beginning, and we have to first form a prior by some non-Bayesian way, so that we can subsequently apply the Bayesian approach?

Yes.

But beware of canonical prior construction. It might sound mathematically appealing but is not automatically realistic from a Bayesian point of view. It is possible a mathematically nice prior actually corresponds to a dumb belief system. For example if you study $X\sim N(\mu,1)$, Jeffrey's prior on $\mu$ is uniform and if about people's average size, this might not be a very realistic system. However with only a few observations, the problem actually disappears quite fast. The choice is not very important.

True problems with prior specification happen in more complicated problems in my opinion. What is important here is to understand what a certain prior says.

First of all, Bayesian approach is often used because you want to include prior knowledge in your model to enrich it. If you don't have any prior knowledge, then you stick to so-called "uninformative" or weekly informative priors. Notice that uniform prior is not "uninformative" by definition, since assumption about uniformity is an assumption. There is no such a thing as a truly uninformative prior. There are cases where "it could be anything" is a reasonable "uninformative" prior assumption, but there are also cases where stating that "all values are equally likely" is a very strong and unreasonable assumption. For example, if you assumed that my height can be anything between 0 centimeters and 3 meters, with all of the values being equally likely a priori, this wouldn't be a reasonable assumption and it would give too much weight to the extreme values, so it could possibly distort your posterior.

On another hand, Bayesian would argue that there is really no situations where you have no prior knowledge or beliefs whatsoever. You always can assume something and as a human being, you're doing it all the time (psychologists and behavioral economists made tons of research on this topic). The whole Bayesian fuss with the priors is about quantifying those preconception and stating them explicitly in your model, since Bayesian inference is about updating your beliefs.

• @AlecosPapadopoulos sorry for not saying what you wanted to hear, but I believe that this is a part of answer for your question. Regarding Q1, obviously assuming uniform prior is not the same as not assuming prior, since you made an assumption. If you don't want to use priors at all, use maximum likelihood or empirical Bayes approach. – Tim Feb 2 at 18:30
• What I "wanted to hear"? As I understand it, when a person poses a question here, it is reasonably expected that the answer will be about the question. There is nothing particular I "wanted to hear" (no priors here too), I just sought answers to specific questions, and my comment was about not seeing in what way your answer addressed my questions. But in your comment I think there is something really relevant: "Empirical Bayes approach"? Can you mention/point to some literature? – Alecos Papadopoulos Feb 2 at 18:58
• @AlecosPapadopoulos empirical Bayes is choosing your priors based on data (i.e. cheating). You can start with Wikipedia or papers by Efron (easily googlable on Google scholar). – Tim Feb 2 at 19:27
• Suppose, you started on a problem on Monday, and had a prior, say standard normal. So, you plug it into your data, run the analysis, learn something. On Tuesday you can't use that prior anymore, because you learned something already. So, you have to plug a different prior, really. So, in strict Bayesian the priors are single use. You literally can run them through the software only ONCE. The moment you get the results, the prior is expired, unless you didn't learn ANYTHING. So in practical sense Bayesian approach is unusable in its pure form, all Bayesians are constantly cheating themselves – Aksakal Feb 2 at 19:57
• @Aksakal But why is invalid to use, on Tuesday, as my new prior, the posterior I obtained on Monday? The way I say it, it is a totally valid sequential procedure. So I do not understand why you write "Bayesians are constantly cheating themselves". – Alecos Papadopoulos Feb 3 at 15:17

question 1 I think the answer is probably no. My reason is we don't really have a definition for "uninformative" except for somehow measuring how far the final answer is from some arbitrarily informative model/likelihood. Many uninformative priors are validated against "intuitive" examples where we already have "the model/likelihood" and "the answer" in mind. We then ask the uninformative prior to give us the answer we want.

My problem with this is I struggle with believing that someone can have a really good, well informed model or model structure for their population, and simultaneously have "no information" about likely and unlikely parameter values for that model. For example using logistic regression, see "A WEAKLY INFORMATIVE DEFAULT PRIOR DISTRIBUTION. FOR LOGISTIC AND OTHER REGRESSION MODELS"

I think the discrete uniform prior is the only one we could reasonably say is the "first-first" prior. But you run into problems of using it, thinking you have "no information", but then suddenly having reactions to "unintuitive" answers (hint: if you don't like a bayesian answer - you might have left information out of the prior or likelihood!). Another problem you run into is getting the discretisation right for your problem. And even thinking of this, you need to know the number of discrete values to apply the discrete uniform prior.

Another property to consider for your prior is the "tail behaviour" relative to the likelihood you are using.

on to question 2

Conceptually, I don't see anything wrong with specifying a distribution without the use of a prior or likelihood. You can start a problem by saying "my pdf is ... and I want to calculate ... wrt this pdf". Then you are creating a constraint for the prior, prior predictive, and likelihood. The bayesian method is for when you have a prior and a likelihood, and you want to combine them into a posterior distribution.

It's probably a matter of being clear on what your probabilities are. Then the argument shifts to "does this pdf/pmf represent what I say it represents?" - which is the space you want to be in I think. From your example, you are saying the single distribution reflects all the available information - there is no "prior" because it's already contained (implicitly) in the distribution you are using.

You can also apply bayes in reverse - what "prior", "likelihood" and "data" gives me the actual prior I am considering? This is one way you can see that a $U (0,1)$ prior for a $Bin(n,p)$ likelihood "looks" like it corresponds to a "posterior" for a $Beta (0,0)$ "prior" with $2$ observations - $1$ from each category.

on the so called blatantly wrong comment

To be honest, I would be very interested to see how any numbet of observation could be used to predict a "statistically independent" observation. As an example, if I tell you I'll generate 100 standard normal variables. I give you 99, and get you to give me your best prediction for 100th. I say you cannot make a better prediction for the 100th than 0. But this is the same you would predict for the 100th if I gave you no data. Hence you learn nothing from the 99 data points.

However, if I tell you that it was "some normal distribution", you can use the 99 data points to estimate the parameters. Then the data are now no longer "statistically independent", because we learn more about the common structure as we observe more data. Your best prediction now uses all 99 data points

• (+1) Thank you for your thoughtful answer. A clarification regarding the "blatantly wrong" assertion: it was made because "learning" (and I am talking about the general meaning of the word) is a much more wider concept than "predicting". If two events are structurally similar, we can learn things related to the one by studying the other, even though they may be statistically independent. You also talk about the "common structure" in your answer, that's all there is to it. – Alecos Papadopoulos Feb 4 at 12:40
• @Alecos Papadopoulos - the point is you cannot learn without making things statistically dependent. Taking my example, what is learnable in scenario 1?Additionally the common structure needs to be unknown, not just present. – probabilityislogic Feb 4 at 21:28
• Commenting on the last sentence of your post, the fact that we can learn something about the common structure as you point out, does not make the random variables involved "statistically dependent". They remain "independent in probability", which is another way to say "statistically independent", a concept that has a very precise meaning mathematically. That they share common characteristics (here, their range is characterized by the same probability distribution), does not make them statistically dependent. – Alecos Papadopoulos Feb 4 at 21:46
• Your phrasing "independent in probability" is not clear to me, and I suspect this is why I disagree with what you are saying. If this is replaced by "conditionally independent" or "exchangeable" then what you say makes sense. I'm also still waiting for something that could be learned from 99 iid standard normal rvs that helps with the 100th (need not be about prediction). – probabilityislogic Feb 14 at 7:31
• @probabilisticlogic "Independent in probability" is an expression that can be found usually in older works, and means what statistical independence means as expressed through distribution functions. The 99 rv's will allow me to learn all sorts of properties, characteristics etc of the 100th, moments, quantiles, you name it. – Alecos Papadopoulos Feb 14 at 17:33

This is only a short remark as addition to the other excellent answers. Often, or at least sometimes, it is somewhat arbitrary (or conventional) what part of the information entering a statistical analysis is called data and which part is called prior. Or, more generally, we can say that information in a statistical analysis comes from three sources: the model, the data, and the prior. In some cases, such as linear models or glm's, the separation is quite clear, at least conventionally.

I will reuse an example from Maximum Likelihood Estimation (MLE) in layman terms to illustrate my point. Say a patient enters a physician's office, with some medical problems that turn out to be difficult to diagnose. This physician hasn't seen something quite similar before. Then, talking with the patient it surfaces some new information: this patient visited tropical Africa quite recently. Then it appears to the physician that this could be malaria or some other tropical disease. But note, that this information is clearly to us data, but at least in many statistical models that could be used, it will enter the analysis in the form of a prior distribution, a prior distribution giving higher probability to some tropical diseases. But we could, maybe, make some (larger), more complete model, where this information enters as data. So, at least in part, the distinction data/prior is conventional.

We are used to, and accept, this convention because of our emphasis on some classes of conventional models. But, in the larger scheme of things, outside the world of stylized statistical models, the situation is less clear.