22
$\begingroup$

I'm confused about how to reconcile the probability of independent events not having anything to do with prior history, but sequences of events do (seemingly) take into account prior history. This question asks a similar question: Probability of independent events given the history. However, having read that, I found I had a very specific confusion about the seeming contradiction between two formulas for probabilities that seem equal to me, but will produce different results based on our understanding of P of sequences versus P of independent events:

(A) P(HHHHH) = 0.03125

(B) P(H | HHHH) = 0.5

Can anyone explain how the left side of both equations, P(HHHHH) and P(H | HHHH) are different.

And does anything change if we shift from a frequentist to bayesian perspective?

$\endgroup$
  • 5
    $\begingroup$ Nothing changes by shifting perspective since those perspectives relate to inverse probability (inference) which is different from these coin flips, which are direct probability. $\endgroup$ – Sextus Empiricus Jun 16 at 18:52
  • 1
    $\begingroup$ Thanks @SextusEmpiricus, that helps me understand the context better. $\endgroup$ – saeranv Jun 16 at 19:05
  • 13
    $\begingroup$ The notation indeed seems a bit confusing. One could interpret P(H|HHHH) to be equal to 1 (as we already know the result of the first flip). It would be more consistent to rewrite it as P(HHHHHH | HHHHH) = 0.5. $\endgroup$ – kemerover Jun 17 at 3:36
  • 4
    $\begingroup$ It may also be relevant to note that P(HHHHH) = P(HHHHH | HHHH) * P(HHHH) by definition of conditional probability (since P(HHHHH | not-HHHH)=0), even without any assumptions about independence of tosses or fairness of the coins. So they can't be equal unless P(HHHH) is 1. $\endgroup$ – Peteris Jun 17 at 9:39

11 Answers 11

53
$\begingroup$

P(HHHHH) is the probability of having five heads in a row. But, P(H|HHHH) means having heads if the last four tosses were heads. In the former, you're at the beginning of the experiment and in the latter one you have already completed four tosses and know the results. Think about the following rewordings:

P(HHHHH): If you were to start the experiment all over again, what would be the probability of having five heads?

P(H|HHHH): If you were to start the experiment but keep restarting it until you got four heads in a row, and then, given that you have four heads, what would be the probability of having the final one as heads?

| cite | improve this answer | |
$\endgroup$
  • 7
    $\begingroup$ Your rewordings here help a lot. The right side of A and B is not the same because we're not accounting for the probability of getting to (the rare) case of four successive heads in B, whereas in A we are. $\endgroup$ – saeranv Jun 16 at 19:28
  • 47
    $\begingroup$ Since a fair coin has no memory, $P(H|HHHH) = P(H)$. So we are actually comparing $P(HHHHH)$ to $P(H)$. Which may make it easier to understand what is going on. $\endgroup$ – Stephan Kolassa Jun 16 at 19:38
36
$\begingroup$

P(HHHHH)

There are 32 possible outcomes from flipping a coin 5 times. Here they are listed:

HHHHH THHHH HTHHH TTHHH HHTHH THTHH HTTHH TTTHH
HHHTH THHTH HTHTH TTHTH HHTTH THTTH HTTTH TTTTH
HHHHT THHHT HTHHT TTHHT HHTHT THTHT HTTHT TTTHT
HHHTT THHTT HTHTT TTHTT HHTTT THTTT HTTTT TTTTT

All of these outcomes are equally likely. So the probability of any one of these sequences is 1/32 = .03125. That's why P(HHHHH) = .03125.

P(H | HHHH)

We are now considering the possible outcomes of a single coin flip, having just observed 4 heads in a row. The only two possible outcomes of this single coin flip, they are of course the following:

H
T

Since the coin flips are assumed independent, the fact that we just observed 4 heads in a row is irrelevant, so this is just the same as considering P(H), the probability of heads for a single toss, regardless of what was just observed. That's why P(H | HHHH) = 0.5.

| cite | improve this answer | |
$\endgroup$
  • 13
    $\begingroup$ Equivalently, P(H|HHHH) can be considered by only considering those 32 outcomes that start with HHHH. That leaves only HHHHH and HHHHT, and so the probability that the 5th flip is heads is 0.5. $\endgroup$ – Calvin Godfrey Jun 17 at 17:42
9
$\begingroup$

Often it's helpful to thing of conditions in terms of information:

$$ \mathbb{P}[H | HHHH] $$

can be read as "The probability of getting Heads, given that I have 4 heads already", i.e., given the information that there are already 4 heads.

Of course, we're told the coin tosses are independent, so this information is not helpful -- the past tosses have nothing to do with the upcoming tosses, i.e., this information tells us nothing about the probability of the upcoming event. Hence (since it's a fair coin), $ \mathbb{P}[H | HHHH] = .5$

We can think of the lack of a condition as being the lack of information, so $ \mathbb{P}[H] $ is "the probability of Heads, with no further information", and

$$ \mathbb{P}[H | HHHH] = \mathbb{P}[H] $$

is a restatement of the above -- the probability of getting heads given the information that we already have 4 heads is the same as the probability of getting heads with no other information.

Lastly we can thus see

$$ \mathbb{P}[HHHHH] $$

As "the probability of 5 heads, with no further information". This means that we don't know the outcome of any tosses yet (since those outcomes would count as information), and there we get our $\mathbb{P}[HHHHH] = \frac1{32}$ -- of all the $2^5$ possible outcomes of 5 tosses (starting from when we don't yet know any outcomes), there is only 1 where all the tosses are H.

| cite | improve this answer | |
$\endgroup$
7
$\begingroup$

The notation that does not index the coin throws and/or their outcomes (and does not even separate the outcomes by commas or signs of intersection) may be confusing. How do we know which coin throw each $H$ refers to in $P(H|HHHH)$ or $P(HHHHH)$? We can often guess, but this is needlessly ambiguous.

Let us index the coin throws and their outcomes by natural numbers. Given that the coin has no memory, it is hopefully clearer why $$ P(H_1|H_1,H_2,H_3,H_4)=1, $$ but $$ P(H_5|H_1,H_2,H_3,H_4)=0.5 $$ and $$ P(H_1,H_2,H_3,H_4,H_5)=0.03125. $$

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

I would suggest you to run a simulation, and view the conditional distribution as apply filter on data.

Specifically, you may

  1. simulation large amount of (say 5 million) coin flips on 5 fair coins
  2. try to find for first 4 coins, the results are HHHH
  3. select a subset of the data by first 4 coins' results are HHHH
  4. check the distribution on the 5th coin.

You may find it is close to 0.5.

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

Fair independent coin

The probability of event a (5th case is heads $H_5$) given event b (already 4 heads $H_4H_3H_2H_1$)

$$\underbrace{P(H_5|H_4H_3H_2H_1)}_{\text {P(a given b)}} = \frac{\overbrace{P(H_5 \& H_4H_3H_2H_1)}^{\text{P(a and b)}}}{\underbrace{P(H_5 \& H_4H_3H_2H_1) }_{\text {P(a and b)}}+\underbrace{P(T_5 \& H_4H_3H_2H_1)}_{\text {P((not a) and b)}}}$$

For a fair coin you have $P(T_5H_4H_3H_2H_1) = P(H_5H_4H_3H_2H_1) = 0.5^5$. And the above equation will be

$${P(H_5|H_4H_3H_2H_1)} = \frac{0.5^5}{0.5^5+0.5^5} = 0.5$$


With a fair coin that is also independent (note we may have $p_{heads}=p_{tails}$ but that does not necessarily mean that the flips are independent), you should get the above result. But that is not the general result.


Unfair coin (or coin with nonindependent flips)

But if the coin is possibly unfair or not independent from flip to flip then this may not be true. Based on some assumed probability distribution for the fairness of the coin you may compute different values for $P(T_5H_4H_3H_2H_1)$ and $P(H_5H_4H_3H_2H_1)$.

In the more general case (the coin is not necessarily fair) you might get that given already four heads, $P(H_5|H_4H_3H_2H_1)>P(H_5)$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Not sure how this is derived. Shouldn't this be: P(H | HHHH) = (P(HHHH | H) * P(H)) / P(HHHH)? $\endgroup$ – saeranv Jun 16 at 19:00
  • 1
    $\begingroup$ @saeranv The case of four heads HHHH will lead in either HHHHH or THHHH. P(HHHH) = P(THHHH) + P(HHHHH). P(HHHH|H) which could better be written less ambiguously as P(HHHH|HHHHH) will equal 1 since you need to have four heads by definition when you have five heads. $\endgroup$ – Sextus Empiricus Jun 16 at 19:08
  • 1
    $\begingroup$ It does P(HHHH) = 0.5^4 = 0.5^5 + 0.5^5 = P(HHHHH) + P(THHHH) $\endgroup$ – Sextus Empiricus Jun 16 at 19:46
  • 1
    $\begingroup$ However P(HHHH|HHHHH) will equal 1. It is a conditional probability. More precisely the probability that you flipped 4 heads when you flipped 5 heads. You are certain to have a the sequence of the first 4 flips to be 4 heads if you have a the first 5 flips equal to 5 heads. $\endgroup$ – Sextus Empiricus Jun 16 at 19:49
  • 1
    $\begingroup$ Ah, I see. So this derivation shows how the P(HHHHH)=0.03125 is accounting for the 0.0625 of P(HHHH) to get P=0.5. Very helpful. $\endgroup$ – saeranv Jun 16 at 19:52
2
$\begingroup$

I see two explainations:

1: (Same as already posted, but specific to fair coin Heads and Tails) Because there are only two possibilities, H and T, P(H|HHHH) is the same as P(HHHHH) / (P(HHHHH) + P(HHHHT)) P(HHHHH) = .5^5 and P(HHHHT) = .5^5, therefore P(H|HHHH) = .5^5/(.5^5+.5^5) = .5^5/(2*(.5^5)) = 1/2

2: P(HHHHH) is the whole story, P(H|HHHH) is just the last chapter.

The probability of getting to HHHHH after 5 flips is (1/2)^5 because each flip has an unconditional probability of 1/2, as stated. The paradox of P(H|HHHH) being represented as (1/2) is related to ignoring the probability of getting into a state of HHHH after 4 flips before looking at the next flip. In conclusion, 1/2 seems surprisingly high of a probability for P(H|HHHH) without considering that the probability of achieving a state of HHHH is low ((1/2)^4) in the first place.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

$P(H|\text{anything}) = P(H) = 0.5$ The probability of the toss of a fair coin being "heads" is half, unconditionally, no matter what other events have occurred previously or at the same time.

The $|$ conditional probability notation $P(A|B)$ primarily expresses the probability of $A$. The $B$ gives the condition which modifies the meaning of $P(A)$ from the perspective of situation $B$ being true.

$P(A|B)$ may be regarded as a macro operator according to this definition:

$$P(A|B) \equiv \frac{P(A\cap B)}{P(B)}$$

Therefore, if we substitute our parameters of interest:

$$P(H|HHHH) \equiv \frac{P(H\cap HHHH)}{P(HHHH)}$$

But $P(H\cap HHHH)$ just means the probability of tossing four heads, and then one more: it means exactly the same thing as $P(HHHHH)$. In fact, $P(HHH...)$ is a shorthand for $P(H\cap H\cap H ...)$. Thus:

$$P(H|HHHH) \equiv \frac{P(H\cap HHHH)}{P(HHHH)} \equiv \frac{P(HHHHH)}{P(HHHH)}$$

It is the probability of tossing five heads, divided by the probability of tossing four heads. Since coin tosses are independent and their probabilities are multiplied together, this is just:

$$\frac{(1/2)^5}{(1/2)^4} = 1/2 = P(H)$$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Take some care with your implicit assumptions. For instance, $P(H\mid H)=1,$ not $1/2,$ demonstrating that "anything" cannot be just anything! $\endgroup$ – whuber Jun 19 at 17:21
  • 1
    $\begingroup$ @whuber That is right. If I almost lay a coin down on the table so that it is face up, and then release it ... given that initial condition, it's virtually certain that it will be heads up when it stops moving. $\endgroup$ – Kaz Jun 19 at 19:27
1
$\begingroup$

It may help to think of these independent events like the individual steps used in climbing a mountain. Each step takes you only one step up the mountain. Yet, although those steps are the same in distance and effort, each subsequent step also results in you being higher up.

How is it that my last step up the mountain can take me all the way to the top, when I am still only stepping one step? Still only expending a fractional amount of energy? Of course it is because all of the prior steps created a situation where one more of exactly the same steps would result in me reaching the top.

In flipping coins, each coin flip has a 50% chance of taking you one flip closer to your goal. That 50% chance, when successful, only ever buys you one more "heads" toward getting 5 in a row. One more step. And it's always a 50% chance. However, if you have already experienced a somewhat rare set of four in a row, that one more step (which is just like the rest) will complete the set of 5 in-a-row... If it comes to pass... Which it will, exactly 50% of the time.

Hope that helps.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

The observations are independent, so the previous draws don’t affect the next draw. Thus P(H) = P(T) = 0.50 if you saw THTHH beforehand or H beforehand or TTTHHH beforehand. What you actually saw last does not affect the next draw. If the previous event is yet unseen, however, you are not dealing with a conditional probability.

That is also true when looking forward. Because each draw does not impact its successor, you have to multiply the probabilities of outcomes. That’s how you get P(TH) = 0.25 but P(T|H) = 0.50.

Conditional probability is the likelihood that something happens given something has already happened — emphasis on past tense.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Just a thought experiment related to the question.

Event 1: If you flip a coin the probability of getting a heads is 0.5 right?

Event 2: At the start of the universe what was the probability that one day you would flip that coin and get that heads? An incredibly small number. Not 0.5 anyway.

Event 1 involved conditional probability even though it wasn't mentioned. The condition was that everything in the universe lined up nicely such that you would flip the coin. Then we start calculating the probability from there.

When you flip a coin the probability of getting heads P(H) could be expressed as P(H|everything that ever happened in the universe up to that point)

Just a thought that crossed my mind when reading your question.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.