Confused about independent probabilities. If a fair coin is flipped 5 times, P(HHHHH) = 0.03125, but P(H | HHHH) = 0.5

Question

I'm confused about how to reconcile the probability of independent events not having anything to do with prior history, but sequences of events do (seemingly) take into account prior history. This question asks a similar question: Probability of independent events given the history. However, having read that, I found I had a very specific confusion about the seeming contradiction between two formulas for probabilities that seem equal to me, but will produce different results based on our understanding of P of sequences versus P of independent events:

(A) P(HHHHH) = 0.03125

(B) P(H | HHHH) = 0.5

Can anyone explain how the left side of both equations, P(HHHHH) and P(H | HHHH) are different.

And does anything change if we shift from a frequentist to bayesian perspective?

Answer 1

P(HHHHH) is the probability of having five heads in a row. But, P(H|HHHH) means having heads if the last four tosses were heads. In the former, you're at the beginning of the experiment and in the latter one you have already completed four tosses and know the results. Think about the following rewordings:

P(HHHHH): If you were to start the experiment all over again, what would be the probability of having five heads?

P(H|HHHH): If you were to start the experiment but keep restarting it until you got four heads in a row, and then, given that you have four heads, what would be the probability of having the final one as heads?

Answer 2

P(HHHHH)

There are 32 possible outcomes from flipping a coin 5 times. Here they are listed:

HHHHH THHHH HTHHH TTHHH HHTHH THTHH HTTHH TTTHH
HHHTH THHTH HTHTH TTHTH HHTTH THTTH HTTTH TTTTH
HHHHT THHHT HTHHT TTHHT HHTHT THTHT HTTHT TTTHT
HHHTT THHTT HTHTT TTHTT HHTTT THTTT HTTTT TTTTT

All of these outcomes are equally likely. So the probability of any one of these sequences is 1/32 = .03125. That's why P(HHHHH) = .03125.

P(H | HHHH)

We are now considering the possible outcomes of a single coin flip, having just observed 4 heads in a row. The only two possible outcomes of this single coin flip, they are of course the following:

H
T

Since the coin flips are assumed independent, the fact that we just observed 4 heads in a row is irrelevant, so this is just the same as considering P(H), the probability of heads for a single toss, regardless of what was just observed. That's why P(H | HHHH) = 0.5.

Answer 3

Often it's helpful to thing of conditions in terms of information:

$$ \mathbb{P}[H | HHHH] $$

can be read as "The probability of getting Heads, given that I have 4 heads already", i.e., given the information that there are already 4 heads.

Of course, we're told the coin tosses are independent, so this information is not helpful -- the past tosses have nothing to do with the upcoming tosses, i.e., this information tells us nothing about the probability of the upcoming event. Hence (since it's a fair coin), $ \mathbb{P}[H | HHHH] = .5$

We can think of the lack of a condition as being the lack of information, so $ \mathbb{P}[H] $ is "the probability of Heads, with no further information", and

$$ \mathbb{P}[H | HHHH] = \mathbb{P}[H] $$

is a restatement of the above -- the probability of getting heads given the information that we already have 4 heads is the same as the probability of getting heads with no other information.

Lastly we can thus see

$$ \mathbb{P}[HHHHH] $$

As "the probability of 5 heads, with no further information". This means that we don't know the outcome of any tosses yet (since those outcomes would count as information), and there we get our $\mathbb{P}[HHHHH] = \frac1{32}$ -- of all the $2^5$ possible outcomes of 5 tosses (starting from when we don't yet know any outcomes), there is only 1 where all the tosses are H.

Answer 4

The notation that does not index the coin throws and/or their outcomes (and does not even separate the outcomes by commas or signs of intersection) may be confusing. How do we know which coin throw each $H$ refers to in $P(H|HHHH)$ or $P(HHHHH)$? We can often guess, but this is needlessly ambiguous.

Let us index the coin throws and their outcomes by natural numbers. Given that the coin has no memory, it is hopefully clearer why $$ P(H_1|H_1,H_2,H_3,H_4)=1, $$ but $$ P(H_5|H_1,H_2,H_3,H_4)=0.5 $$ and $$ P(H_1,H_2,H_3,H_4,H_5)=0.03125. $$

Answer 5

I would suggest you to run a simulation, and view the conditional distribution as apply filter on data.

Specifically, you may

1. simulation large amount of (say 5 million) coin flips on 5 fair coins
2. try to find for first 4 coins, the results are HHHH
3. select a subset of the data by first 4 coins' results are HHHH
4. check the distribution on the 5th coin.

You may find it is close to 0.5.

Answer 6

Fair independent coin

The probability of event a (5th case is heads $H_5$) given event b (already 4 heads $H_4H_3H_2H_1$)

$$\underbrace{P(H_5|H_4H_3H_2H_1)}_{\text {P(a given b)}} = \frac{\overbrace{P(H_5 \& H_4H_3H_2H_1)}^{\text{P(a and b)}}}{\underbrace{P(H_5 \& H_4H_3H_2H_1) }_{\text {P(a and b)}}+\underbrace{P(T_5 \& H_4H_3H_2H_1)}_{\text {P((not a) and b)}}}$$

For a fair coin you have $P(T_5H_4H_3H_2H_1) = P(H_5H_4H_3H_2H_1) = 0.5^5$. And the above equation will be

$${P(H_5|H_4H_3H_2H_1)} = \frac{0.5^5}{0.5^5+0.5^5} = 0.5$$

---

With a fair coin that is also independent (note we may have $p_{heads}=p_{tails}$ but that does not necessarily mean that the flips are independent), you should get the above result. But that is not the general result.

---

Unfair coin (or coin with nonindependent flips)

But if the coin is possibly unfair or not independent from flip to flip then this may not be true. Based on some assumed probability distribution for the fairness of the coin you may compute different values for $P(T_5H_4H_3H_2H_1)$ and $P(H_5H_4H_3H_2H_1)$.

In the more general case (the coin is not necessarily fair) you might get that given already four heads, $P(H_5|H_4H_3H_2H_1)>P(H_5)$

Answer 7

I see two explainations:

1: (Same as already posted, but specific to fair coin Heads and Tails) Because there are only two possibilities, H and T, P(H|HHHH) is the same as P(HHHHH) / (P(HHHHH) + P(HHHHT)) P(HHHHH) = .5^5 and P(HHHHT) = .5^5, therefore P(H|HHHH) = .5^5/(.5^5+.5^5) = .5^5/(2*(.5^5)) = 1/2

2: P(HHHHH) is the whole story, P(H|HHHH) is just the last chapter.

The probability of getting to HHHHH after 5 flips is (1/2)^5 because each flip has an unconditional probability of 1/2, as stated. The paradox of P(H|HHHH) being represented as (1/2) is related to ignoring the probability of getting into a state of HHHH after 4 flips before looking at the next flip. In conclusion, 1/2 seems surprisingly high of a probability for P(H|HHHH) without considering that the probability of achieving a state of HHHH is low ((1/2)^4) in the first place.

Answer 8

$P(H|\text{anything}) = P(H) = 0.5$ The probability of the toss of a fair coin being "heads" is half, unconditionally, no matter what other events have occurred previously or at the same time.

The $|$ conditional probability notation $P(A|B)$ primarily expresses the probability of $A$. The $B$ gives the condition which modifies the meaning of $P(A)$ from the perspective of situation $B$ being true.

$P(A|B)$ may be regarded as a macro operator according to this definition:

$$P(A|B) \equiv \frac{P(A\cap B)}{P(B)}$$

Therefore, if we substitute our parameters of interest:

$$P(H|HHHH) \equiv \frac{P(H\cap HHHH)}{P(HHHH)}$$

But $P(H\cap HHHH)$ just means the probability of tossing four heads, and then one more: it means exactly the same thing as $P(HHHHH)$. In fact, $P(HHH...)$ is a shorthand for $P(H\cap H\cap H ...)$. Thus:

$$P(H|HHHH) \equiv \frac{P(H\cap HHHH)}{P(HHHH)} \equiv \frac{P(HHHHH)}{P(HHHH)}$$

It is the probability of tossing five heads, divided by the probability of tossing four heads. Since coin tosses are independent and their probabilities are multiplied together, this is just:

$$\frac{(1/2)^5}{(1/2)^4} = 1/2 = P(H)$$

Answer 9

It may help to think of these independent events like the individual steps used in climbing a mountain. Each step takes you only one step up the mountain. Yet, although those steps are the same in distance and effort, each subsequent step also results in you being higher up.

How is it that my last step up the mountain can take me all the way to the top, when I am still only stepping one step? Still only expending a fractional amount of energy? Of course it is because all of the prior steps created a situation where one more of exactly the same steps would result in me reaching the top.

In flipping coins, each coin flip has a 50% chance of taking you one flip closer to your goal. That 50% chance, when successful, only ever buys you one more "heads" toward getting 5 in a row. One more step. And it's always a 50% chance. However, if you have already experienced a somewhat rare set of four in a row, that one more step (which is just like the rest) will complete the set of 5 in-a-row... If it comes to pass... Which it will, exactly 50% of the time.

Hope that helps.

Answer 10

The observations are independent, so the previous draws don’t affect the next draw. Thus P(H) = P(T) = 0.50 if you saw THTHH beforehand or H beforehand or TTTHHH beforehand. What you actually saw last does not affect the next draw. If the previous event is yet unseen, however, you are not dealing with a conditional probability.

That is also true when looking forward. Because each draw does not impact its successor, you have to multiply the probabilities of outcomes. That’s how you get P(TH) = 0.25 but P(T|H) = 0.50.

Conditional probability is the likelihood that something happens given something has already happened — emphasis on past tense.