How subjective are prior beliefs?

Question

I find the term 'prior-beliefs' to be a little bit vague, what is acceptable to class as a prior belief in Bayesian analysis? For example, I could not look at any data and decide to myself "I believe the mean of the population is going to be about 60units", whether this is based on intuition or even lack of care. Are there any criteria for what counts as a 'solid' prior belief?

Answer 1

A classical way to explain subjective probabilities is that if you assign a probability of $p$ to an event $A$, this means that you are willing to pay up to $p$ for a gamble in which you win 1 if $A$ takes place and nothing otherwise, and up to $1-p$ for a gamble in which you win if $A^c$ happens (i.e., when assigning prior probabilities you don't know whether an opponet will force you to pay for $A$ or for $A^c$, and therefore you have an incentive to choose $p$ high enough but not too high). This refers to observable events $A$, i.e., predictions of future outcomes implied by parametric distributions rather than values of the unobservable parameters themselves, so that bets can be paid out based on future observations.

Ideally there would be somebody available to bet against you, i.e., to take your money for some or all of the sets $A$ on which you have offered bets, and pay you out afterwards (obviously "all sets" isn't possible in case there are infinitely many sets as is for example the case if you specify a continuous prior distribution).

If this were indeed the case, you'd have a strong motivation to think as hard as you can about your choice of prior. It would be fully subjective, as it refers to you personally, you'd personally pay and win. Nobody can stop you from coming up with some random nonsense, but you yourself would be the one to be hurt, so you better don't.

As long as you play the game on your own, without requirement to convince others, in principle there is no need to give explicit justifications of your prior, but of course you'd like to think through things properly (and may well decide to go by some justification that is good enough to present to others) because you don't want to lose.

Now reality and science don't really work like that; it's an idealisation. Two things are different. (a) You won't normally find anyone who takes your bets (although such things have been arranged in certain exceptional situations) and (b) science is about communication, it's about finding out things for "humankind as a whole" rather than just for you personally.

(a) could in principle be handled by imagining the idealised situation and behaving "as if", i.e., choosing your prior assignments imagining that you will have to pay for gambles as indicated above against an anonymous opponent who can take your money for either $A$ or $A^c$ for any possible event. The problem here is that you don't lose or win anything real, so the incentive is weaker and people may not try quite as hard to think things through as if their real money were at stake. Note also that in many situations (as for example any problem involving continuous distributions) the number of possible events $A$ is very large or infinite, and thinking things through in a way that you have enough "coverage" of all things that can happen can be very hard (even if you have real betting opponents; these will have a hard time, too, deciding what bets to take, of course). Of course in reality you may be confronted with a real decision problem and in one way or another you will later have to face the consequences, even though not exactly in the way the subjective Bayesian setup above assumes - how good an incentive that is depends on the real situation of course.

Regarding (b), it is a legitimate concern to ask why science, striving for objectivity, should care about subjective probabilities at all. However, thinking about how to assign priors in an objective way, in all but the most simplistic situations you will find that decisions are required where there is no clear objective quantification. Most knowledge doesn't come in a form that can be directly and uniquely translated into prior probabilities, so any researcher who wants to do this needs at some point to bring in personal judgement. Even the idea to use "informationless" priors isn't really convincing because (i) in most situations it is controversial how to do this, and (ii) it can be seen as misleading and against the Bayesian spirit to choose an informationless prior if in fact information is available (if your personal money were at stake, you'd for sure not be happy about the prospect to ignore available information). We have written about the tension between desired objectivity and necessary subjectivity in statistics here (Gelman & Hennig, "Beyond Subjective and Objective in Statistics").

So at the end of the day there will need to be some personal judgement and prior choices will be subjective to at least some extent.

The key thing here in my view is communication and what I see as the aim of science, namely reaching knowledge on which general consensus is possible. This is driven (among other things) by transparency and the possibility for criticism. This means that if you want to produce valuable scientific results based on Bayesian reasoning and prior choices, you need to motivate and explain your priors in a transparent way so that others can agree or criticise them. People may then either agree to your choices and results, or everyone involved can learn and improve based on the arguments that come up in a model discussion (there is also certain work about constructing consensus priors that refer to a group of people rather than an individual; in any case there may hopefully be some kind of convergence of views through open discussion).

The good thing about this is that this provides an additional incentive to think through your prior well, and to present it in an optimally convincing way, having in mind that any detail is up for scrutiny.

Standard literature about Bayesian statistics presents a number of principles to give some orientation how to choose priors in a reasonable way, so there is some guidance on valid argumentation. This is not "algorithmic"; I haven't see any recipe that would guide you in a straightforward way to a prior that you should use based on a small number of simple decisions; the variety of situations is just too big, but it helps for sure. (I can't check it right now but I'm pretty sure I have seen such principles listed in de Finetti's Theory of Probability and Bernardo and Smith's Bayesian Theory.) The ultimate test is in any case what the scientific community makes of the proposed priors.

I should add that unfortunately this is not how things in reality often work out. We rarely find deep and comprehensive discussions of prior choices. Often in the literature supposedly informationless priors are proposed, with the authors probably hoping that these can be taken as "objective" and that criticism is avoided (but see above). Sometimes we see the statement that "prior choice doesn't matter because with enough observations everyone will converge to essentially the same posterior", but apart from the fact that the number of available observations may not be big enough, this statement assumes certain prior choices such as exchangeability and compatible basic model setups that are not themselves covered by such logic. So very often we see very little if any convincing justification of priors that are used in practice, which means that we should be skeptical! This has certainly to do with the difficulty and complexity of the task, and also with the drive to appear objective even where objectivity clearly doesn't reach far enough.

Answer 2

Are there any criteria for what counts as a 'solid' prior belief?

Priors are not arbitrary distributions. One must be able to defend or justify their selection of priors. I would argue that objectivity is really just agreement on subjective experiences (for what are we but imperfect observers all agreeing we observe the same things), so in my opinion all one really needs to do is to have people agree that the priors reflect some existing knowledge or data.

Additionally, one should be comfortable with interpreting results on a spectrum of reliability as opposed to "solid" or not, and realize that just as priors vary from person to person, so too will the standard for acceptance with little resistance.

As an example for justifying one's priors,here is a fake conversation modelled after one I often have when describing my models.

Me: ...and for this parameter, we use a normal prior with a mean of log(3.5) and a standard deviation of 0.1.

Them: Why those priors? How did you arrive at them

Me: The author of the paper from which these priors are based works for the manufacturer of the drug under study. The continuously publish studies on the pharmacokinetics of the drug. The paper to which I reference is a systematic review of the pharmacokinetics of the drug. That paper references estimated parameters from smaller samples. We took the mean and standard deviation of those samples for use in our prior.

Them: I see, so your prior selection is partially informed by past empirical work. How do we know these are the right prior distributions?

Me: There is no "right" prior distribution, and had that study received a slightly different sample we may have used a slightly different prior. What we can say is that when we simulate from these priors, we get values of the parameter which are consistent with published data and with our collaborators domain knowledge on the subject.

Them: And do you anticipate your conclusions would change substantially if those priors were different?

Me: There is understandably some sensitivity to the prior selection, but we performed a prior sensitivity analysis wherein we varied our prior selection and compared posterior estimates. We find the posterior is most sensitive to priors on that parameter as opposed to this parameter. We're hoping to gather more information on that parameter so as to make our choice of prior more justifiable and aligned with domain knowledge.

Answer 3

There are certainly criteria such as whether it is based on prior studies, how many prior studies, how close those are to your situation, and so on. You can adjust your prior distribution for degree of certainty or strength of belief, with one end being uninformative priors, which will have minimal impact on the posterior.

I don't know of any systematic way to combine those criteria into something that can be used to figure out how informative to make your priors, but I am no expert on that. However, remember that you will need to write up the results and that includes writing up your priors. Your priors have to be justifiable.