To a Bayesian, does a trick coin with two heads have 50% chance of flipping heads if they don't know that it has two heads?

Question

And to expand on this idea, would uncertainty and complexity be the same thing to a Bayesian? For example, games with much randomness like poker and games that are more complexity based like chess -- would we treat what makes them "interesting" (I know there are other interesting aspects to these games) as the same sort of thing?

Answer 1

Note that for a Bayesian*, their prior distribution on a parameter represents their prior knowledge/beliefs about the parameter(s) of interest.

The prior on P(Head) would normally be a distribution with support on [0,1].

It might well be centered at $\pi_H=0.5$, or it might not. It might have mixture of a spike at $\pi_H=0.5$ plus some continuous distribution around it, or it might be pure-continuous. It could be a single spike at $\pi_H=\frac12$.

It's perfectly possible for an individual Bayesian to have a prior distribution that has $0$ density on a possible value for the parameter; they can exclude the actual case, if under their knowledge or belief, such a value is not possible - a Bayesian can have false beliefs. However, a wiser Bayesian might seek to not exclude the possibility of a coin that has heads (or tails) on both sides and place a tiny amount of the prior on such possibilities. A suspicious Bayesian may well put a substantial amount of their prior on such cases.

In short, yes, a Bayesian's prior belief can fail to accord any probability to a situation that actually occurs.

For example, games with much randomness like poker and games that are more complexity based like chess -- would we treat what makes them "interesting" (I know there are other interesting aspects to these games) as the same sort of thing?

I'm not sure I follow what sort of model you're supposing** - what the response is and what information might be in the model.

In general there's little reason to suppose similar models for the two unless your model is extremely abstract.

---

* at least for most varieties of Bayesian

** nor how it relates to the title question, to be honest

Answer 2

Coins don't have probabilities. The mathematical model of a coin flip has probabilities. It depends on what "if they don't know that it has two heads" is supposed to mean in the context of the mathematical model. Does the model permit a coin with two heads? If so, what are the probabilities of the knowledge or non-knowledge? What are the probabilities of two-head coins (or other non-fair coins) in the context of coins available for flipping in that particular experiment?

If the equipment for a statistical experiment does not match the model proposed for the experiment, that is not a problem of Bayesian or other statistics, like it isn't a problem of a runner's training state when they don't manage to run 100m in the time they would expect because the distance or the clock are rigged or the track is a mud fest.

Answer 3

If the Bayesian believes that just about all coins are fair, yes, the Bayesian would use a prior that emphasizes $0.5$. However, if the Bayesian starts flipping the coin and gets a billion straight heads, the posterior will (or at least should) start to favor a high probability of flipping heads, unless the prior puts all density on $0.5$ (or no density toward the coin favoring heads).

Remember that frequentist and Bayesian inferences are used to use the known data to infer something about the unknown, such as the probability parameter of a Bernoulli distribution. Once you show the Bayesian the coin for her to see that it has heads on both sides, she no longer has to make an inference about the probability. She knows the population distribution.

Answer 4

Suppose I give you the choice between the following two deals:

• Deal 1: I have a biased coin which lands heads with probability 55%. I flip it. If it comes up heads, you win a million bucks.

• Deal 2: The same, but instead of my trick coin we use the next random coin we find lying in the street, regardless of whether that coin has heads on one side, both sides, or neither side.

If you prefer Deal 1, that suggests that you agree with the Bayesian that the probability of the unknown coin flipping heads is < 55%.

(The Bayesian won't necessarily think the probability of getting heads is exactly 50% - e.g. they might believe there are more double-headed coins than double-tailed coins in the world.)

Answer 5

To a Bayesian, does a trick coin with two heads have 50% chance of flipping heads if they don't know that it has two heads?

This sentence is a bit weird. How can a statistician contemplate the probability of a certain coin with two heads if they don't know it has two heads? The contemplated coin with it's properties is something that is assumed to be known. We use expressions like $P(\text{heads | trick coin})$, the probability of heads given that it is known that the coin is a trick coin with two heads.

What is possible is that a statistician is thinking about the probability of heads for a some unknown coin that is in reality a trick coin but they don't know that it is a trick coin. Yet, that statistician will have a model for a hypothetical trick coin that is:

A trick coin with two heads has 100% chance of flipping heads.

Note the Bayes rule:

$$ P(\text{trick coin | heads}) \propto P(\text{heads | trick coin}) \cdot P(\text{trick coin})$$

The statistician may not need to know whether some coin is a trick coin (ie. the prior is $0 < P(\text{trick coin}) < 1$).

But, the statistician will model the outcomes of a trick coin as $P(\text{heads | trick coin}) = 1$ and not $P(\text{heads | trick coin}) = 0.5$.

Answer 6

Most of the time you should ignore all the philosophizing about subjective states of knowledge and just use a weakly informative prior. The reason why is that all too often the beliefs you're considering including in your prior aren't really knowledge at all; they're assumptions. Take, for example, the belief that the probability that a coin toss will come up heads 50% of the time. That's an assumption. It's a pretty good assumption much of the time, but we don't know it to be true for any given experiment. We can easily imagine situations in which it's false, such as if the person who gave us the coin is a con artist who is actively trying to deceive us.

The thing to keep in mind is, if you include too little information in your prior, then the worst that happens is your estimates are a little more uncertain than they might otherwise have been, which is not a big deal. If, on the other hand you include too much information, such as a bunch of assumptions that happen to be false, then you will get an answer that is wildly incorrect, which is disastrous. Therefore, unless you have a specific reason to limit the types of solutions you are willing to consider, with a weakly informative prior you are much more likely to get a useful result.

In the case of our coin example, that means that for some arbitrary coin, where you have no particular knowledge of how it was chosen, you should use for $p$ (the probability of a toss coming up heads) something like a uniform prior, $p \sim (\text{Beta}(1,1))$, or Jeffreys' prior, $p \sim (\text{Beta}(1/2, 1/2))$. These are both uninformative priors that will quickly be overwhelmed by observations from actually tossing the coin. Neither one is likely to lead you astray. You could try to encode some idea that fair coins and two-headed or two-tailed coins are more common than weighted coins, but the gain from doing so is marginal, and if we're being honest, we don't really know that to be true in any particular situation; it's just an assumption.

Like any rule, this one has its exceptions. To stick with our coin example, if you know the coin to have been chosen arbitrarily from the vault of a reputable bank, then perhaps you might choose a prior centered on $p = 1/2$. However, even then I wouldn't make the prior very strong. What's the point of doing the experiment if you are going to pre-specify the answer?

A better example is when you are forced to make predictions without having the opportunity to run an experiment to determine $p$ empirically. In such cases your prior is all you have to go on. In a scenario like that, if you're lucky, you might get inspect the coin to see that it does indeed have both a head and a tail, and then you might just assume $p = 0.5$. It's not much to go on, but it's what you've got. My best advice for these situations is to avoid getting into them. If you can't do that, my second best advice, if you can't do that, is to get comfortable with being wrong.

Answer 7

The Bayesian would choose a prior distribution for p, the probability of flipping heads, which should be any valid probability distribution on [0, 1].

As long as a good prior is used, as the Bayesian observes and collects data, the Bayesian's posterior distribution will approach (heads ~ 1.0, tails ~ 0.0)