12
$\begingroup$

And to expand on this idea, would uncertainty and complexity be the same thing to a Bayesian? For example, games with much randomness like poker and games that are more complexity based like chess -- would we treat what makes them "interesting" (I know there are other interesting aspects to these games) as the same sort of thing?

$\endgroup$
9
  • 7
    $\begingroup$ For a subjectivist Bayesian probabilities are subjective, so it will depend on which Bayesian you ask. In any case for a subjectivist the probability is not a property of the coin itself but rather a property of the person who thinks about the coin, and the information they have. $\endgroup$ Commented Sep 24, 2023 at 15:36
  • 5
    $\begingroup$ Isn't there a hidden assumption here? That frequentists are smarter than Bayesians, so a frequentist would look at both sides of the coin. What do you suppose would happen if this assumption were incorrect? $\endgroup$ Commented Sep 24, 2023 at 20:24
  • 2
    $\begingroup$ I haven't heard anyone claim that uncertainty and complexity are the same thing. For example consider rolling a $32^64$ sided die. There are more possible outcomes than there are chess positions, but still this situation is very simple. $\endgroup$ Commented Sep 24, 2023 at 22:33
  • 3
    $\begingroup$ Suppose I told you that I had a collection of coins, some fair, some double-headed and some double-tailed; I know how many of each but you do not. What is your belief about the probability of Heads if you pick a coin at random and flip it? $\endgroup$
    – Henry
    Commented Sep 25, 2023 at 13:14
  • 3
    $\begingroup$ Your question appears to be about definitions and notations more than about the distinction between frequentist and Bayesian approaches; and you might be interested in the answers to this similar question: What is the probability of a fair coin flip that has already occurred? $\endgroup$
    – Stef
    Commented Sep 25, 2023 at 14:43

7 Answers 7

15
$\begingroup$

Note that for a Bayesian*, their prior distribution on a parameter represents their prior knowledge/beliefs about the parameter(s) of interest.

The prior on P(Head) would normally be a distribution with support on [0,1].

It might well be centered at $\pi_H=0.5$, or it might not. It might have mixture of a spike at $\pi_H=0.5$ plus some continuous distribution around it, or it might be pure-continuous. It could be a single spike at $\pi_H=\frac12$.

It's perfectly possible for an individual Bayesian to have a prior distribution that has $0$ density on a possible value for the parameter; they can exclude the actual case, if under their knowledge or belief, such a value is not possible - a Bayesian can have false beliefs. However, a wiser Bayesian might seek to not exclude the possibility of a coin that has heads (or tails) on both sides and place a tiny amount of the prior on such possibilities. A suspicious Bayesian may well put a substantial amount of their prior on such cases.

In short, yes, a Bayesian's prior belief can fail to accord any probability to a situation that actually occurs.

For example, games with much randomness like poker and games that are more complexity based like chess -- would we treat what makes them "interesting" (I know there are other interesting aspects to these games) as the same sort of thing?

I'm not sure I follow what sort of model you're supposing** - what the response is and what information might be in the model.

In general there's little reason to suppose similar models for the two unless your model is extremely abstract.


* at least for most varieties of Bayesian

** nor how it relates to the title question, to be honest

$\endgroup$
19
$\begingroup$

Coins don't have probabilities. The mathematical model of a coin flip has probabilities. It depends on what "if they don't know that it has two heads" is supposed to mean in the context of the mathematical model. Does the model permit a coin with two heads? If so, what are the probabilities of the knowledge or non-knowledge? What are the probabilities of two-head coins (or other non-fair coins) in the context of coins available for flipping in that particular experiment?

If the equipment for a statistical experiment does not match the model proposed for the experiment, that is not a problem of Bayesian or other statistics, like it isn't a problem of a runner's training state when they don't manage to run 100m in the time they would expect because the distance or the clock are rigged or the track is a mud fest.

$\endgroup$
3
  • 2
    $\begingroup$ Nice clarifying distinction in the opening couple of sentences there. Would be worth the +1 even on its own. $\endgroup$
    – Glen_b
    Commented Sep 25, 2023 at 10:51
  • 1
    $\begingroup$ This is a much more appropriate answer than the other two. $\endgroup$
    – Stef
    Commented Sep 25, 2023 at 14:44
  • $\begingroup$ Then i guess baysians cant actually asign probablitys to anything occuring. (because then they would be confusing reality with their model of reality) $\endgroup$ Commented Sep 30, 2023 at 15:10
9
$\begingroup$

If the Bayesian believes that just about all coins are fair, yes, the Bayesian would use a prior that emphasizes $0.5$. However, if the Bayesian starts flipping the coin and gets a billion straight heads, the posterior will (or at least should) start to favor a high probability of flipping heads, unless the prior puts all density on $0.5$ (or no density toward the coin favoring heads).

Remember that frequentist and Bayesian inferences are used to use the known data to infer something about the unknown, such as the probability parameter of a Bernoulli distribution. Once you show the Bayesian the coin for her to see that it has heads on both sides, she no longer has to make an inference about the probability. She knows the population distribution.

$\endgroup$
1
  • $\begingroup$ Bayesians should not believe just about all coins are fair, though might start by believing most coins in circulation are not a long way from being fair, while retaining Cromwell's rule that they could be wrong: the coin could be substantially biased or could be double-headed or double tailed. They mick also believe that trick coins are more likely to be less fair than coins in circulation $\endgroup$
    – Henry
    Commented Sep 26, 2023 at 13:12
3
$\begingroup$

Suppose I give you the choice between the following two deals:

  • Deal 1: I have a biased coin which lands heads with probability 55%. I flip it. If it comes up heads, you win a million bucks.

  • Deal 2: The same, but instead of my trick coin we use the next random coin we find lying in the street, regardless of whether that coin has heads on one side, both sides, or neither side.

If you prefer Deal 1, that suggests that you agree with the Bayesian that the probability of the unknown coin flipping heads is < 55%.

(The Bayesian won't necessarily think the probability of getting heads is exactly 50% - e.g. they might believe there are more double-headed coins than double-tailed coins in the world.)

$\endgroup$
1
$\begingroup$

To a Bayesian, does a trick coin with two heads have 50% chance of flipping heads if they don't know that it has two heads?

This sentence is a bit weird. How can a statistician contemplate the probability of a certain coin with two heads if they don't know it has two heads? The contemplated coin with it's properties is something that is assumed to be known. We use expressions like $P(\text{heads | trick coin})$, the probability of heads given that it is known that the coin is a trick coin with two heads.

What is possible is that a statistician is thinking about the probability of heads for a some unknown coin that is in reality a trick coin but they don't know that it is a trick coin. Yet, that statistician will have a model for a hypothetical trick coin that is:

A trick coin with two heads has 100% chance of flipping heads.

Note the Bayes rule:

$$ P(\text{trick coin | heads}) \propto P(\text{heads | trick coin}) \cdot P(\text{trick coin})$$

The statistician may not need to know whether some coin is a trick coin (ie. the prior is $0 < P(\text{trick coin}) < 1$).

But, the statistician will model the outcomes of a trick coin as $P(\text{heads | trick coin}) = 1$ and not $P(\text{heads | trick coin}) = 0.5$.

$\endgroup$
0
$\begingroup$

Most of the time you should ignore all the philosophizing about subjective states of knowledge and just use a weakly informative prior. The reason why is that all too often the beliefs you're considering including in your prior aren't really knowledge at all; they're assumptions. Take, for example, the belief that the probability that a coin toss will come up heads 50% of the time. That's an assumption. It's a pretty good assumption much of the time, but we don't know it to be true for any given experiment. We can easily imagine situations in which it's false, such as if the person who gave us the coin is a con artist who is actively trying to deceive us.

The thing to keep in mind is, if you include too little information in your prior, then the worst that happens is your estimates are a little more uncertain than they might otherwise have been, which is not a big deal. If, on the other hand you include too much information, such as a bunch of assumptions that happen to be false, then you will get an answer that is wildly incorrect, which is disastrous. Therefore, unless you have a specific reason to limit the types of solutions you are willing to consider, with a weakly informative prior you are much more likely to get a useful result.

In the case of our coin example, that means that for some arbitrary coin, where you have no particular knowledge of how it was chosen, you should use for $p$ (the probability of a toss coming up heads) something like a uniform prior, $p \sim (\text{Beta}(1,1))$, or Jeffreys' prior, $p \sim (\text{Beta}(1/2, 1/2))$. These are both uninformative priors that will quickly be overwhelmed by observations from actually tossing the coin. Neither one is likely to lead you astray. You could try to encode some idea that fair coins and two-headed or two-tailed coins are more common than weighted coins, but the gain from doing so is marginal, and if we're being honest, we don't really know that to be true in any particular situation; it's just an assumption.

Like any rule, this one has its exceptions. To stick with our coin example, if you know the coin to have been chosen arbitrarily from the vault of a reputable bank, then perhaps you might choose a prior centered on $p = 1/2$. However, even then I wouldn't make the prior very strong. What's the point of doing the experiment if you are going to pre-specify the answer?

A better example is when you are forced to make predictions without having the opportunity to run an experiment to determine $p$ empirically. In such cases your prior is all you have to go on. In a scenario like that, if you're lucky, you might get inspect the coin to see that it does indeed have both a head and a tail, and then you might just assume $p = 0.5$. It's not much to go on, but it's what you've got. My best advice for these situations is to avoid getting into them. If you can't do that, my second best advice, if you can't do that, is to get comfortable with being wrong.

$\endgroup$
0
$\begingroup$

The Bayesian would choose a prior distribution for p, the probability of flipping heads, which should be any valid probability distribution on [0, 1].

As long as a good prior is used, as the Bayesian observes and collects data, the Bayesian's posterior distribution will approach (heads ~ 1.0, tails ~ 0.0)

$\endgroup$
3
  • $\begingroup$ Welcome to Cross Validated! Prior distribution of what? $\endgroup$
    – Dave
    Commented Sep 25, 2023 at 21:12
  • $\begingroup$ The Bayesian would be more likely to choose a dispersed prior distribution which, even if it might be centred on a probability of Heads of $\frac12$, would still be supported on a range from $0$ to $1$ $\endgroup$
    – Henry
    Commented Sep 26, 2023 at 13:14
  • $\begingroup$ Oops, good point! The prior should be a distribution for p $\endgroup$
    – Jack
    Commented Oct 1, 2023 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.