If the Bayesian probability is not a belief, what is it?

Question

In this blog by William Briggs, who seems to be a prolific lecturer/writer on pop-statistics, he condemns the "Bayesian Metaphor" which is essentially referring to Bayesian probability as belief.

Quoting from a post on vampires, “In Bayesian inference, you start with some initial beliefs (called ‘Bayesian priors’ or just ‘priors’), and then you ‘update’ them as you receive new evidence.”

This is the standard metaphor, and it’s not so much wrong as unhelpful, misleading, and restricting. The metaphor derives from Bayes’s rule (details which can be looked up anywhere) and which gives a formula which on the right-hand-side is supposed to be an element representing “prior beliefs.” The formula itself is correct, as most math is. But because math is correct does not mean that it means what you think it means.

In my opinion, the prior being a "belief" is not a metaphor, but it is a veritable reflection of Bayesian probability. For instance, Bayesians can probabilistically quantify events that frequentists cannot such as the location of a quantum particle. I understand the distinction between Bayes' Rule and Bayesian Statistics: the former is a probabilistic law that applies to frequentist as well as Bayesian inference as well; the latter deals with a Bayesian interpretation of probability. He goes on to say:

This is wrong because there is no such thing as “Pr(Y)” or “Pr(X)”. These objects do not exist. Numbers can be put in their place and the equation can be made to work out, but it is the step of putting numbers in that is wrong. There is no such thing as an unconditional probability, so we can never write without error “Pr(Y)” or “Pr(X)”. Instead, we should write e.g. Pr(Y|W) or Pr(X|W), where W is the knowledge we start with, i.e. our real prior (knowledge).

And I think at this point--that this post doesn't seem to have anything to do with Bayesian statistics at all. Like a frequentist putting uncertainty bounds on uncertainty bounds, it seems the Bayesian analogue is conditioning on conditioning.

Is this blog sound reasoning? Is there actually a distinction between "belief" and "knowledge" when specifying a prior? If a conditional density exists, such as $Pr(Y|W)$ then a marginal density exists as well $Pr(Y) = \int Pr(Y|W)Pr(W)$?

Answer 1

Perhaps what the author was getting as is distinguishing between the posterior distribution and an individual's posterior distribution.

Suppose we knew as a fact that some of our parameters followed a given distribution before the data was distributed. Then based on those parameters, we observed some data conditional on those parameters and we knew the form of that conditional distribution. We could then apply Bayes theorem and know the probability distribution of said parameters, conditional on the data we saw.

On the other hand, traditional Bayesian statistics tells us that our uncertainty can be a prior distribution. That's fine and I think most modern statisticians have zero problem with that concept. However, it's sometimes swept under the rug that the final output is still conditional on the original prior, and generally speaking there's no reason to believe the next person will have the same prior as you.

To illustrate, consider the following R/psuedocode to demonstrate:

# Simulate mean first, then simulate data
simData = function(n = 10){
  # Simulate uniform(0,1)
  rand_unif = runif(1, min = 0, max = 1)
  # Mu is either -1 or 1 with probability 0.5
  if(rand_unif > 0.5){ mu = 1 }
  else{ mu = -1 }
  #Simulate
  output = rnorm(n, mu = mu, sd = 1)
  return(output)
}

Now if I showed you this code in advance and run the function, you can easily compute the probability that mu = 1 in the given simulation, despite never seeing it. This will unquestionable be correct (up to numerical error).

On the other hand, suppose you know the code up to

# Mu is some value <cliffsFavoriteNumber>
simData = function(n = 10){
  mu = cliffsFavoriteNumber
  #Simulate
  output = rnorm(n, mu = mu, sd = 1)
  return(output)
}

People other than me don't know the distribution on cliffsFavoriteNumber. Maybe Joe says "I don't think Cliff would choose a big number and it's gotta be a positive integer, so I'd say the prior for mu is a rounded exponential". Bob thinks "Cliff loves multiples of 9, so I'd say uniform prior on 9, 18, ..., 81". The computations they can compute from there are not mathematically wrong, but they condition on different people's prior believes and as such end up with very different answers. Thus, it's a little misleading for Joe to say something like "The posterior distribution is ...". It's more precise to say something like "Conditional on believing that the prior distribution on mu is a rounded exponential, the posterior distribution will be ...".

Answer 2

The problem is that you W can't be integrated out. Taken all the knowledge you have, W, we form our prior, p(Y|W). And with this, we can use Bayes theorem to get our posterior p(Y|XW). W just tags along with all our updates, but we don't actually know p(W). If we did have the probability of our prior beliefs we could integrate W out, but that probably would itself be conditioned on its own prior beliefs. No matter what there are always prior beliefs that can't be integrated out.

I would like to note that many places don't include background knowledge as part of Bayes theorem, but it is there. Bayes theorem as properly expressed should read

$p(y|xw)=\frac{p(x|yw)p(y|w)}{p(x|w)}$

See how w just tags along?

Answer 3

If the prior accurately reflects your beliefs about the parameter and you are willing to make the assumption that the model for the data is correct, then the posterior is the rational way to update your beliefs after seeing the data.

The assumption that your model for the data is correct is implicit and sometimes not discussed, though it doesn't make sense to use conditional probability notation to make the assumption. Admittedly though, this would draw more attention to the assumptions being made that are sometimes not discussed explicitly.

Similarly, $P(X)$ is understood to mean the marginal probability of the data averaged over your prior beliefs.

Saying $P(Y)$ does not exist is odd, though it is true that it can be difficult to construct $P(Y)$ in such a way that it does accurately reflect your prior beliefs.