Bayesian Analysis in the Absence of Prior Information?

Question

I have always wondered - how confident do researchers tend to be in their "prior" information when deciding to create statistical models using a Bayesian Approach vs. a Frequentist Approach?

As an example : Suppose a team of researchers has some data relating to the healthcare industry - let's say that they are interested in fitting regression models to this data. As such, they are familiar with the scientific background of the data they are working with, but they are unsure what kinds of priors they can place on the parameters of these regression models.

I saw this post over here (What is an "uninformative prior"? Can we ever have one with truly no information?) in which the idea of an "Uninformative Prior" is provided - but apart from the idea of the Uninformative Prior, I can not seem to find more information on this topic.

As such, is the following idea true? When we don't have a lot of confidence on information that can be used to construct Bayesian Priors - is it generally better to stick to Frequentist Approaches instead of "guessing" which Priors will better fit the data and the choice of model?

Answer 1

If you have no information, you can use “uninformative” priors. Those priors aim to bring as little information as possible, but as you already learned from the What is an "uninformative prior"? Can we ever have one with truly no information? thread, the name is a little bit misleading because even such priors bring some information to the model. That is why more modern recommendation would be to pick a weakly informative prior (centered on something, but very uncertain).

On another hand, using a frequentist model does not mean making no assumptions: you may still assume things like Gaussian likelihood, a linear relationship between the variables, or using regularized regression you are using implicit priors on the parameters, etc. Even if you wanted to make as few assumptions as possible and used a nonparametric model, you would be making some assumptions. For example, say that you would pick $k$NN regression that just averages among "similar" observations. Still, you need to decide on a similarity metric to define what "similar" means and you need to somehow pick the hyperparameter $k$. With Bayesian models, you make additional assumptions by choosing priors, but other approaches also make assumptions.

But, can you have truly no information about something? Say a meteor hits the planet Earth and brings us a new virus from space. A Bayesian statistician needs to build a model on it. They know nothing about space viruses. Hopefully, they know a lot about viruses from Earth, many scientists also did a lot of educated guesses on what extraterrestrial life forms could be and how they could be similar or different to the life on Earth, etc. The scientist in fact has a lot of prior knowledge and assumptions that they could use to come up with priors.

Answer 2

I think that the posterior inherits meaning from the prior, which implies that if the prior is meaningless, so is the posterior.

Now there are different varieties of Bayesians when it comes to interpreting probabilities, i.e., assigning meaning to priors (and posteriors). Most (but not all) Bayesian interpretation of probabilities is epistemic, meaning that probabilities formalise a state of belief or knowledge rather than a data generating mechanism that exists in reality (as frequentists do).

Subjectivist Bayesians traditionally state that the prior should formalise your personal prior belief. This particularly means that probabilities do not model data, and data cannot contradict the prior (as observing data later cannot change your belief before data). Note that this applies to both the parameter prior and the likelihood in standard Bayesian setups in which there is a parametric likelihood and a prior on the parameters. The original literature (de Finetti and Ramsey) postulates that you basically have prior beliefs about everything that can be formalised as prior distribution. One way to operationlise that is to ask you to bet on how the data will turn out before you see it, implying that your are forced to offer bets on all kinds of possible outcomes. Existing prior information will make you expect certain outcomes more than others, so that you should put higher prior probability on these. If you don't have much information, you need to spread out probability so that everything that is possible has enough share in the overall distribution that data ultimately can push the posterior there if there is a clear message from the data.

The thing is that in real data analysis hardly anyone is forced to bet in advance, and for sure not on enough events to determine the prior completely. In fact there is some literature about imprecise probabilities in which you are not forced to bet but where the few bets (or specifications, if proper bets are not involved) you are willing to make determine "upper" and "lower" (prior, but later also posterior) probabilities. A similar thing happens in Bayesian sensitivity analysis - you may not be willing to specify a precise prior, but rather a range of different priors may look appropriate to you, and you can run Bayesian analyses with all of these and see how much results differ.

In reality, you may have some information, but this information may not translate readily into prior probabilities such as "the probability for $\theta$ to be between -2 and 2 is 85%"; you may think it's likely, but whether that translates into 70, 85, or 92% may not be so clear to you, and is certainly not enforced by the information itself. The subjectivist approach is that you should basically then decide how to place your bets (i.e., if you are ready to pay 85 for a possible win of 100 in case this happens, or rather not). What I think is misleading about this idea is that subjectivists seem to think that there exists some "true" personal prior that can be "found out" in this way, whereas I think that you are rather forced to make it up out of more or less thin air if you want to do a Bayesian analysis. Obviously sensitivity analysis can show you, when things run well (which often means that you need to have a lot of data), that your final conclusions may not be all that different given different choices that look equally plausible to you, and although you make some choices that don't seem that well founded, this doesn't affect the conclusions much. Unfortunately, in much published real Bayesian analysis, disappointingly little care is invested into motivating the prior and into exploring sensitivity in case of alternative choices, as these are hard tasks and everyone (not only the frequentists!) wants statistics to be easy. If indeed the posterior inherits meaning from the prior, there isn't much to expect from the resulting posteriors.

Objectivist Bayesians would not like to think of probabilities as made up by individuals forced to bet, but the objectivist idea rather is that the prior should only formalise "objective" external information that can be verified and agreed upon. Otherwise non-informative priors should be used. Unfortunately this is of little help in the situation that there is prior information that doesn't come in form of objectively verifiable probabilities. Almost all prior information is of this kind. In practice objectivist analysis therefore either becomes subjective at least to some extent (J. Berger says in some papers that so-called objective Bayes is not really objective but rather points to the attitude that we should try as hard as we can to reach an ultimately unattainable ideal), or people use non-informative priors despite in fact having some information, which deprives the Bayesian approach of one of its major benefits. (On top of that, even what is called non-informative priors will often carry some subtle information that may harm analyses if not consciously acknowledged and used.)

There is also the possibility to actually interpret probabilities in an aleatory way (i.e., non-epistemic, referring to data generating processes in the world; some say "frequentist). A. Gelman occasionally advertises Bayesian statistics in this way. The data is then in fact informative about the prior, and if the data contradict it, it may be changed, although this violates the Bayesian concept of coherence. For setting up a convincing informative model this may make sense, but posterior probabilities should not be interpreted in a naive fashion such as "after data the probability that the true $\theta$ is between -2 and -1.5 is 98.5%", because the model within which that "true $\theta$" is defined is an idealisation and may be changed with more information coming in (which by the way should also be kept in mind for frequentist analyses); also the potential of changing the prior with the data may lead to overprecision, i.e., credible intervals may be too small because the prior that in Bayesian analysis is meant to be fixed before data has been adapted to the data (the same issue occurs in many frequentist analyses).

Personally I think there are big issues with any kind of statistical approach; Bayesians often tell frequentists off for many issues that in some way or another they actually face themselves (for example there is the accusation of "adhockery", frequentists are accused of adapting their approach in non-principled and irregular ways to the situation at hand, but if Bayesians want to learn and adapt the prior rather than being constrained by whatever their first prior choice implies, they violate one of their own major principles).

Setting up a prior basically means to put weights on the parameter (and observation) space, which will then influence analyses. There are situations in which there are good reasons to do this, particularly if there is some information that is meant to influence the analysis (but then of course this needs to be translated into prior probabilities and this is hard). I'm not keen though on doing Bayesian analysis for the sake of it, so from my personal view, if there is no prior information that you want to use to do this kind of reweighting, I'd be happy to do a frequentist analysis. (Note that there are still various ways how prior information can influence your analysis; the idea that relevant information should always and only be used to determine prior probabilities for outcomes and parameters is nonsense.)

I will also acknowledge that there may be reasons to want epistemic probabilities rather than frequentist ones (particularly if the frequentist ideal of unlimited identical repetition of experiments doesn't seem convincing), in which case a Bayesian approach even with supposedly non-informative priors can be preferred. However, the postulate that there are unique true epistemic probabilities looks just as questionable to me as the postulate of infinitely repeatable experiments, resulting in the existence of true frequentist probabilities. Also, be careful to prevent your prior from bringing in some implicit informations/implications that you don't want.

Answer 3

When we don't have a lot of confidence on information that can be used to construct Bayesian Priors - is it generally better to stick to Frequentist Approaches instead of "guessing" which Priors will better fit the data and the choice of model?

In my opinion, the full power of a Bayesian analysis is unleashed when a priori information is available, although how to translate this prior information into a distribution is far from trivial and there may be several different ways to do it.

Indeed, building a prior from extra experimental (or expert opinions) data is, to some extent, akin to choosing the statistical model. Thus the inferential task for a subjective Bayesian is heavier than that of a frequentist statistician. The power, however, is that we can, at least in principle, include expert opinion or extra experimental information in our inferential conclusions, information that in a frequentist approach would have been otherwise discarded.

In absence of information, however, we also can find several compelling reasons why a Bayesian approach is still useful. In the linked post you find many of them, and I summarise them as

• lead to inferential procedures with good frequentist properties (probability coverage, unbiasedness of the MAP)
• default priors tailored to the parameter of the model at hand that are guaranteed to impact the posterior as less as possible.

Notice the bold default, which seems to be the most suitable adjective. Indeed, we cannot call them non-informative since "informativeness" is relative to the measure of information used and may depend on the scale of the parameter.

Among these compelling reasons, I find two of them that are particularly relevant. Firstly, some of these default priors have been discovered and used by frequentists in order to adjust their inference that otherwise would have been inaccurate or useless.

For instance, Firth(1993) showed that the Jeffreys prior in exponential families expressed in the canonical form leads to a maximum a posteriori (MAP) with reduced bias. Furthermore, and more importantly for frequentists, he proposed a weight function for frequentists which solves the problem of perfect separation in logistic regression.

However, the most compelling argument in the defence of default priors, such as the Jeffreys, is invariance under one-to-one reparametrizations. (personal communication by J. Berger in an O'Bayes meeting some years ago). And I totally agree with this. Indeed, if we think invariance to parametrization is a good thing, as most frequentists do, then that's the ultimate defence for the use of non-informative or default prior.