# Confusion about Bayesian statistics. Does the probability for heads change from .5 to 1, after observing heads?

P(A): The coin has a 50 percent chance of being Heads. P(A|X): You look at the coin, observe a Heads has landed, denote this information X, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

I have seen this example many times. However, it does not make senses to me. Given that I know a fair coin would be 50/50 chance. When I see a head landed first, I would naively guess that the next one to be Tail. So then when should I update my belief that the probability of Head should be 1.0?

If I were a Frequentist then I would have known the events are independent so I continue to assign 0.5 probability to both Head and Tail. Is that correct?

• Could you give us more context on this? What is the exact description of the problem?
– Tim
Commented Feb 4, 2020 at 7:21
• I'm trying to understand why Bayesian assigns a higher probability to something that one has seen. If I follow that logic, then I will continue assign 1.0 probably to Head if it has come up 3 times in a row already? Shouldn't I start doubting myself and start lowering the probability for Head? Commented Feb 4, 2020 at 7:34
• I'm asking about details, because, I agree, this doesn't make much sense, so maybe you omitted some important details?
– Tim
Commented Feb 4, 2020 at 7:40
• Not sure how much details you want but this is an example from the first chapter of camdavidsonpilon.github.io/… Commented Feb 4, 2020 at 7:43
• I'm re-reading the example. It does make sense to me now since I have already observed the coin and know the answer. The probability being assigned is for the current coin not future ones. Thanks for challenging me :-) Commented Feb 4, 2020 at 7:53

"denote this information X, and trivially assign probability 1.0 to Heads and 0.0 to Tails."

You assign a probability to the frequency, the property how often the coin will flip heads/tails, and not to individual events that have occurred with the coin.

Even when an event has already occurred we may assign a probability to it (the coin) that differs from that single event. Yes, you could say, when I have won a lottery then in hindsight my chances to win that lottery were 1.0 (it was written in the stars), and you might argue that in a deterministic world every single coin flip is fixed to 0 or 1. But, that's not the way to think about it.

In a deterministic world nothing may be truly random in principle. But while God knows exactly how the dice roll (although that is debated), we do not know all information, and in practice we need to work with probability. (you could say that probability is an expression of lack of information)

We have no way to compute all the factors, influences of wind, variable rotation speed each flip, spinning on the floor, different starting positions, etc. in order to know whether the coin flips head or tails. Therefore we model the coin as a random variable. Even afterwards, when the coin has already landed and we know how the (in principal) deterministic event unfolded, we can still treat it as a result that was from the practical point of view a random event.

"When I see a head landed first, I would naively guess that the next one to be Tail."

This is the Gambler's fallacy. If the coin is fair (50/50) then it is just as probably to flip heads-heads as heads-tails, so after you flipped heads you should you expect with equal probability tails as heads (if the coin is fair).

However, in relation to Bayesian statistics, you may not be certain whether the coin is fair and have some knowledge about distributions of fair/unfair coins. In that case you have heads-heads occurring slightly more often than heads-tails (because not all coins are fair and there is a slight chance to encounter coins that fall relatively more often on the same side than mixed). Thus, in fact, if a coin lands heads then we should actually expect the next flip to be slightly more probable to land heads as well (although in most practical settings this is only slight). Thus it is the opposite from "I would naively guess that the next one to be Tail".

In most practical settings (with coins) this is only slight, but you could create examples where it is more obvious. For instance , imagine the case where you have a mixture of fair and unfair coins for which the probability of an unfair coin is reasonable high.

This makes me think of an interresting didactical method to teach students learn Bayesian statistics. As a sort of practical course, let them play gambling games but with dice among which some predetermined fraction of the dice are slightly unfair but the students do not know which dice are unfair. To spice it up the outcome will determine the mark of the student.

For a practical example that illustrates the updating of probabilities see: Probability of a two-headed coin given a few sample flips? (and there are many others of this type: https://www.google.com/search?q=urn+fair+unfair+coins+site:math.stackexchange.com)

P(A): The coin has a 50 percent chance of being Heads. P(A|X): You look at the coin, observe a Heads has landed, denote this information X, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

Given that I have already looked at the coin and know the answer (i.e. X), I would just assign probability P(A|X) of 1.0 to the current coin.

You seemed to have found the answer by yourself, so just for context, let's add some formality for you to understand it better. The distribution for coin flips is Bernoulli, so the likelihood is

$$f(x|p) = p^x \, (1-p)^{1-x}$$

where $$x$$ is tails encoded as $$0$$, or heads encoded as $$1$$, and $$p \in [0, 1]$$ is probability of observing heads. In frequentist setting you rely only on what you saw in the data, without making any out-of-data assumptions. In frequentist setting you would be maximizing the likelihood of seeing your result

$$\hat p_\text{ML} = \underset{p}{\operatorname{arg\,max}} \; p^x \, (1-p)^{1-x}$$

For example, if you observed $$X=1$$, then you can easily check by yourself (e.g. you can plot it) that this function has the highest value for $$p=1$$. This is how, given the data alone, you "assign" $$\hat p_\text{ML} = 1$$, since you observed $$x/n = 1/1$$ heads in the sample of size $$n=1$$.

In Bayesian setting, you additionally are allowed (even required!) to use a prior to describe your a priori assumptions about $$p$$, and you would be maximizing the posterior probability

$$\hat p_\text{MAP} = \underset{p}{\operatorname{arg\,max}} \; f(x|p)\, f(p)$$

This function does not have to have the maximum at $$p=1$$, as it would depend also on the prior $$f(p)$$. With flat prior $$f(p) \propto 1$$ it would have same maximum, as with maximum likelihood, but with other prior the results may be different (check for example how does it behave with different "uninformative" priors). Here we take into consideration also the prior beliefs about the possible distribution of probabilities $$p$$, and given the data, we update the belief.