How do Bayesians deal with the fact that the treatment of fixed parameters as random variables can lead to inappropriate subjective probabilities?

Question

Let's assume the tossing of an unfair coin is modeled by a random variable X taking the values head and tail. You know that the objective probability of the coin showing up head is either $p=0.4$ or $p=0.6$ and that no other value is possible. Further, you have reason to believe that $p=0.6$ is very likely to be true.

Following the Bayesian methodology, you could express your belief by the (subjective) prior distribution

$$\begin{align} P(p=0.6)&=0.9 \\ P(p=0.4)&=0.1\text{,} \end{align}$$

hence treating $p$ as a random variable. Once you do this, marginalization in combination with Lewis's principle implies that your subjective probability for the coin showing up head should be

$$P(X=\text{head})=0.6 \times 0.9+0.4 \times 0.1=0.58\text{,}$$

which is rather inappropriate even as subjective probability, because after all you know that $p$ can either take the value $0.4$ or $0.6$ and hence your subjective probability should either be $0.4$ or $0.6$.

I would like to know how Bayesians respond to that issue. Has this issue been discussed theoretically, and what is the Baysian position towards it?

---

@Arya McCarthy: Thank you for your answer. I think you are right that Baysians interpret P(X=head)=0.58 as degree of belief, that heads will show up. My question is why does this makes sense, and according to which criterion? I feel this boils down to the justification of the prior. Imagine a Bayesian also has the same prior distribution for other coins as well and that the objective probability of getting a coin with p=0.6 is actually 0.9 and the objective probability of getting a coin with p=0.4 is actually 0.1. If in an experiment, the Baysian selects a series of coins at random and tosses them, $P(X=\text{heads})=0.58$ would be justified, because in this two stage random experiment (first select a coin and then toss it) the overall relative frequency of head will roughly speaking converge to the expected value, which is $0.58$. Hence, roughly speaking, a long run strategy based on $P(X=\text{head})=0.58$ would actually be successful in the real world and this seems to me as a good criterion for P(X=head)=0.58 to make sense, - actual long run success considering similar situations. So how do you arrive at a prior, that roughly reflects these two objective probabilities? You could collect a sample and calculate frequentist estimates. If this is not possible, you should somehow try to guess what these two objective probabilities in a two stage experiment really are. Only then can you interpret $P(X=\text{head})=0.58$ as your subjective believe about the long run frequency in a two stage experiment and can argue that you believe this leads to the best long run strategy. That being said, in my initial example, if I had to make a single bet, I personally would base my action on $p=0.6$, because I am not interested in the long run here. This does not mean $P(X=\text{head})=0.58$, which reflects the long run relative frequency if my prior also does.

Answer 1

Expanding my comment into an answer: I don't see this as inappropriate at all. To call it inappropriate is analogous to saying that the the expected value of a die being 3.5 is inappropriate—after all, a 3.5 can never come up!

It might be less confusing to use a less loaded letter than $p$ for the coin's parameter. I'll call it $f$.

You seem concerned that the true $f$ cannot be 0.58. But the Bayesian never claims that the coin's parameter $f$ is 0.58. They're saying that this is their belief about $p(\text{heads})$. The belief incorporates uncertainty about $f$.

Don't forget—you're not betting on the value of $f$. You're betting on the outcome of the sampled coin flip. Believing that $p(\text{heads})$ is 0.58 lets you hedge your bets. In fact, de Finetti explained the concept of subjective probabilities in the context of betting odds. It's exactly what you're considering here: rather than treating $p(\text{heads})$ as 0.6 or 0.4, you use your uncertainty about $f$ to alter how confidently you'd bet on heads.

Answer 2

• What you refer to as $P(X=\text{head})$ is in fact an expected value $E(p)$ of a prior distribution that you've chosen for $p$. You noticed correctly that the distribution can take only two values $0.4$ and $0.6$, so $0.58$ is an impossible result for such distribution. There is nothing wrong with this. Bernoulli distribution can take only two values $0$ and $1$, and its expected value is a real number in the unit interval. The expected value of a discrete distribution, like binomial, or Poisson can be a real number, no matter that those are distributions for integer-valued things. This has nothing to do with Bayesian statistics, it's how random variables and expected values work.

• It’s worth pointing out that the expected value of the prior distribution for $p$ is not the “subjective probability”, it’s just a numerical summary of the prior distribution that you assumed for the parameter $p$. This summary may, or may not make sense to describe the distribution. Your prior is the whole distribution, not its expected value.
  If your prior probability distribution is

  $$ \begin{align} P(p=0.6)&=0.9 \\ P(p=0.4)&=0.1\text{,} \end{align} $$

  and based on this you need to make a single bet, then you could simply bet on $0.6$ as it has a higher probability. There is no reason why you would bet for the impossible $0.58$ average. Expected value just tells you what is $p$ on average, nothing more than this.

• This choice of the prior for $p$ is rather strange, but I understand that this is a made-up example. As you noticed, the distribution can take only two values. If you plug in the prior into the Bayes theorem

  $$ P(A|B) = \frac{P(B|A)\,P(A)}{P(B)} $$

  the result would also need to be defined in terms of those two values. The more usual choice of a prior for probability $p$ would be a distribution over the whole range of possible values for probabilities, e.g. uniform over $(0, 1)$, or beta distribution. Such distributions assign non-zero probabilities to all the possible values of $p$, so you could update the prior with the data (through the likelihood), and come up with an estimate. Usually, you have no way of knowing for certain that the parameter is a particular real number, so both such prior makes more sense, and problems, as described by you, don't arise.

• If you collect the data, the more data you have, the closer the result would be to what you observed in the data. Given that you used a reasonable prior (that assigns non-zero probabilities to the possible values of the parameter), this methodology enables you to estimate the distribution of possible values of the parameters given the data and the prior.

• What is probability? Bruno de Finetti once said that "probability does not exist". Probability is an abstract, mathematical concept. There are no "probabilities" in the wild. If you toss a coin, an unlimited number of different factors make it hard to predict. Each of those factors alone can be described in terms of laws of physics, but together they form a chaotic system. We assign a probability to simplify all those factors to a single number, but there is no "probability" that is responsible for the result of the coin toss.

  Bayesians reason in terms of subjective probabilities, the probability is a number between zero and one that measures how much I believe in something. If I needed to make a bet, the number would help me quantify the amount of the bet that would be reasonable based on my assumptions.

Answer 3

This is not really an objection to Bayesian analysis at all --- it is an objection to the rules of probability theory. If you have an event $\mathcal{E}$ (e.g., a head occurring on a coin) and a discrete parameter $p$ giving the conditional probability of the event, then the law of total probability states that:

$$\mathbb{P}(\mathcal{E}) = \sum \mathbb{P}(\mathcal{E}|p) \pi(p) = \sum p \cdot \pi(p).$$

As can be seen from this equation, the marginal probabilty of the event is a convex combination of the conditional probabilities given by the possible values of the parameter $p$. Your question asserts that the marginal probability of the event should be equal to one of the possible parameter values, but that is not always true. In your example, there is nothing wrong with the conclusion that the marginal probability of a head is $0.58$ --- that is the value that follows from the stipulated prior for $p$ and the likelihood function for the outcome of the coin.

As to a response, Bayesians would have the same response as any other practitioners who use probabilty theory as the basis for statistics. If you are unwilling to accept the law of total probability theory then you are rejecting one or more of the axioms of probability, and the onus would then be on you to stipulate an alternative theory of probability to take its place.