2
$\begingroup$

Let's assume the tossing of an unfair coin is modeled by a random variable X taking the values head and tail. You know that the objective probability of the coin showing up head is either $p=0.4$ or $p=0.6$ and that no other value is possible. Further, you have reason to believe that $p=0.6$ is very likely to be true.

Following the Bayesian methodology, you could express your belief by the (subjective) prior distribution

$$\begin{align} P(p=0.6)&=0.9 \\ P(p=0.4)&=0.1\text{,} \end{align}$$

hence treating $p$ as a random variable. Once you do this, marginalization in combination with Lewis's principle implies that your subjective probability for the coin showing up head should be

$$P(X=\text{head})=0.6 \times 0.9+0.4 \times 0.1=0.58\text{,}$$

which is rather inappropriate even as subjective probability, because after all you know that $p$ can either take the value $0.4$ or $0.6$ and hence your subjective probability should either be $0.4$ or $0.6$.

I would like to know how Bayesians respond to that issue. Has this issue been discussed theoretically, and what is the Baysian position towards it?


@Arya McCarthy: Thank you for your answer. I think you are right that Baysians interpret P(X=head)=0.58 as degree of belief, that heads will show up. My question is why does this makes sense, and according to which criterion? I feel this boils down to the justification of the prior. Imagine a Bayesian also has the same prior distribution for other coins as well and that the objective probability of getting a coin with p=0.6 is actually 0.9 and the objective probability of getting a coin with p=0.4 is actually 0.1. If in an experiment, the Baysian selects a series of coins at random and tosses them, $P(X=\text{heads})=0.58$ would be justified, because in this two stage random experiment (first select a coin and then toss it) the overall relative frequency of head will roughly speaking converge to the expected value, which is $0.58$. Hence, roughly speaking, a long run strategy based on $P(X=\text{head})=0.58$ would actually be successful in the real world and this seems to me as a good criterion for P(X=head)=0.58 to make sense, - actual long run success considering similar situations. So how do you arrive at a prior, that roughly reflects these two objective probabilities? You could collect a sample and calculate frequentist estimates. If this is not possible, you should somehow try to guess what these two objective probabilities in a two stage experiment really are. Only then can you interpret $P(X=\text{head})=0.58$ as your subjective believe about the long run frequency in a two stage experiment and can argue that you believe this leads to the best long run strategy. That being said, in my initial example, if I had to make a single bet, I personally would base my action on $p=0.6$, because I am not interested in the long run here. This does not mean $P(X=\text{head})=0.58$, which reflects the long run relative frequency if my prior also does.

$\endgroup$
12
  • 2
    $\begingroup$ Could you explain what you're trying to ask by "what sense does this make?" $\endgroup$
    – whuber
    May 13, 2021 at 18:34
  • 8
    $\begingroup$ I see no problem with 0.58 as a subjective probability, it is simply taking into account uncertainty over the two possible choices $\endgroup$
    – bdeonovic
    May 13, 2021 at 20:13
  • 2
    $\begingroup$ You'd base it on the first one. (A lot of Bayesian reasoning actually was first described in the context of betting odds.) $\endgroup$ May 13, 2021 at 23:48
  • 4
    $\begingroup$ It might be less confusing to use a less loaded letter than $p$ for the coin's parameter. I'll call it $f$. The asker seems concerned that the true $f$ cannot be 0.58. But the Bayesian never claims that the coin's parameter $f$ is 0.58. They're saying that this is their belief about $p(\text{heads})$. $\endgroup$ May 14, 2021 at 0:48
  • 3
    $\begingroup$ Editing questions is not meant for discussion with other users. Please do not use it like this. Few edits to improve questions are fine, but not frequent edits. $\endgroup$
    – Tim
    May 18, 2021 at 6:50

3 Answers 3

4
$\begingroup$

Expanding my comment into an answer: I don't see this as inappropriate at all. To call it inappropriate is analogous to saying that the the expected value of a die being 3.5 is inappropriate—after all, a 3.5 can never come up!

It might be less confusing to use a less loaded letter than $p$ for the coin's parameter. I'll call it $f$.

You seem concerned that the true $f$ cannot be 0.58. But the Bayesian never claims that the coin's parameter $f$ is 0.58. They're saying that this is their belief about $p(\text{heads})$. The belief incorporates uncertainty about $f$.

Don't forget—you're not betting on the value of $f$. You're betting on the outcome of the sampled coin flip. Believing that $p(\text{heads})$ is 0.58 lets you hedge your bets. In fact, de Finetti explained the concept of subjective probabilities in the context of betting odds. It's exactly what you're considering here: rather than treating $p(\text{heads})$ as 0.6 or 0.4, you use your uncertainty about $f$ to alter how confidently you'd bet on heads.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your answer. I think you are right that Baysians interpret P(X=head)=0.58 as degree of believe, that head will show up. See my longer post below. $\endgroup$
    – Jan
    May 16, 2021 at 9:09
6
$\begingroup$
  • What you refer to as $P(X=\text{head})$ is in fact an expected value $E(p)$ of a prior distribution that you've chosen for $p$. You noticed correctly that the distribution can take only two values $0.4$ and $0.6$, so $0.58$ is an impossible result for such distribution. There is nothing wrong with this. Bernoulli distribution can take only two values $0$ and $1$, and its expected value is a real number in the unit interval. The expected value of a discrete distribution, like binomial, or Poisson can be a real number, no matter that those are distributions for integer-valued things. This has nothing to do with Bayesian statistics, it's how random variables and expected values work.

  • It’s worth pointing out that the expected value of the prior distribution for $p$ is not the “subjective probability”, it’s just a numerical summary of the prior distribution that you assumed for the parameter $p$. This summary may, or may not make sense to describe the distribution. Your prior is the whole distribution, not its expected value.
    If your prior probability distribution is

    $$ \begin{align} P(p=0.6)&=0.9 \\ P(p=0.4)&=0.1\text{,} \end{align} $$

    and based on this you need to make a single bet, then you could simply bet on $0.6$ as it has a higher probability. There is no reason why you would bet for the impossible $0.58$ average. Expected value just tells you what is $p$ on average, nothing more than this.

  • This choice of the prior for $p$ is rather strange, but I understand that this is a made-up example. As you noticed, the distribution can take only two values. If you plug in the prior into the Bayes theorem

    $$ P(A|B) = \frac{P(B|A)\,P(A)}{P(B)} $$

    the result would also need to be defined in terms of those two values. The more usual choice of a prior for probability $p$ would be a distribution over the whole range of possible values for probabilities, e.g. uniform over $(0, 1)$, or beta distribution. Such distributions assign non-zero probabilities to all the possible values of $p$, so you could update the prior with the data (through the likelihood), and come up with an estimate. Usually, you have no way of knowing for certain that the parameter is a particular real number, so both such prior makes more sense, and problems, as described by you, don't arise.

  • If you collect the data, the more data you have, the closer the result would be to what you observed in the data. Given that you used a reasonable prior (that assigns non-zero probabilities to the possible values of the parameter), this methodology enables you to estimate the distribution of possible values of the parameters given the data and the prior.

  • What is probability? Bruno de Finetti once said that "probability does not exist". Probability is an abstract, mathematical concept. There are no "probabilities" in the wild. If you toss a coin, an unlimited number of different factors make it hard to predict. Each of those factors alone can be described in terms of laws of physics, but together they form a chaotic system. We assign a probability to simplify all those factors to a single number, but there is no "probability" that is responsible for the result of the coin toss.

    Bayesians reason in terms of subjective probabilities, the probability is a number between zero and one that measures how much I believe in something. If I needed to make a bet, the number would help me quantify the amount of the bet that would be reasonable based on my assumptions.

$\endgroup$
2
  • $\begingroup$ “Your choice of a prior is rather bad”—the asker indicates that 0.4 and 0.6 are the only possible values. $\endgroup$ May 15, 2021 at 13:51
  • $\begingroup$ @AryaMcCarthy good point. I corrected the wording. $\endgroup$
    – Tim
    May 15, 2021 at 19:49
4
$\begingroup$

This is not really an objection to Bayesian analysis at all --- it is an objection to the rules of probability theory. If you have an event $\mathcal{E}$ (e.g., a head occurring on a coin) and a discrete parameter $p$ giving the conditional probability of the event, then the law of total probability states that:

$$\mathbb{P}(\mathcal{E}) = \sum \mathbb{P}(\mathcal{E}|p) \pi(p) = \sum p \cdot \pi(p).$$

As can be seen from this equation, the marginal probabilty of the event is a convex combination of the conditional probabilities given by the possible values of the parameter $p$. Your question asserts that the marginal probability of the event should be equal to one of the possible parameter values, but that is not always true. In your example, there is nothing wrong with the conclusion that the marginal probability of a head is $0.58$ --- that is the value that follows from the stipulated prior for $p$ and the likelihood function for the outcome of the coin.

As to a response, Bayesians would have the same response as any other practitioners who use probabilty theory as the basis for statistics. If you are unwilling to accept the law of total probability theory then you are rejecting one or more of the axioms of probability, and the onus would then be on you to stipulate an alternative theory of probability to take its place.

$\endgroup$
9
  • $\begingroup$ Thank you for your answer. It is only Bayesians, who treat the fixed constant p as a random variable (frequentists do not). It is only this treatment, which makes it possible to apply the law of total probability to the situation. A frequentist would not agree with the equation you presented. I am not questioning the axioms of probability here, only the treatment of fixed constants as random variables. $\endgroup$
    – Jan
    May 15, 2021 at 17:20
  • $\begingroup$ @Jan but, as noticed correctly by Ben, there is nothing "Bayesian" about your example. It is just a probability distribution and you are calculating the expected value of it, that appears outside the possible values of the random variable. Same thing applies to Bernoulli distribution, so you could argue that if Frequentist estimates the mean of the coin tosses to be 0.513, the number does not make sense. $\endgroup$
    – Tim
    May 15, 2021 at 19:58
  • 1
    $\begingroup$ @Jan: As Tim notes, this is something that also happens in non-Bayesian applications. Many statistical paradigms use the law of total probability, so if you are unwilling to accept values computed from this rule, that is something that will affect more than just Bayesian analysis. $\endgroup$
    – Ben
    May 16, 2021 at 9:22
  • $\begingroup$ As I said, it is only Bayesians, who treat the fixed constant $p$ as a random variable, i.e. as a function $p: \Omega \rightarrow [0,1]$, and only then does $p=0.6$ represents an event, i.e. a subset of $\Omega=\{head,tail\}$, i.e. $\{p=0.6\}=p^{-1}(0.6)=\{\omega \in \Omega : p(\omega)=0.6\}$. The law of total probability applies only to events or subsets of $\Omega$ in this case and not to constants. In the described single stage experiment, no frequentists would agree with the application of the law of total probability. $\endgroup$
    – Jan
    May 16, 2021 at 10:40
  • 2
    $\begingroup$ @Jan this example has nothing to do with Bayesian statistics, it does not even involve Bayes theorem. It uses basic laws of probability. This is how probability theory works. Average human has between one and two legs, this is how expected values work nonetheless that the results does not have much to do with human biology. $\endgroup$
    – Tim
    May 16, 2021 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.