41
$\begingroup$

Let's say we know that A is independent of B, or mathematically:

$$P(A|B) = P(A)$$

Then how come we can't say the following is necessarily true: $$P(A|B,C) = P(A|C)$$

If the outcome of B doesn't have an effect on the outcome of $A$, then why would the outcome of $B$ AND $C$ have an effect on $A$ that is different than the effect of the outcome of just $C$?

Is there perhaps a simple counterexample to illustrate this? For example, let's say $A$ and $B$ are the event of picking an ace of spades from a deck of cards (each has their own deck of cards).

$\endgroup$
1
  • 1
    $\begingroup$ In words: "For more than two events, pairwise independence does not imply mutual independence." $\endgroup$
    – Stef
    Oct 11, 2022 at 8:46

6 Answers 6

84
$\begingroup$

My favourite example is a chessboard: if you pick a point uniformly then the row, column, and colour are pairwise independent.

Suppose A is colour, B is row, and C is column. Then P(A="white"|B)=0.5 and P(A="white"|C)=0.5 for any value of B, but P(A="white"|B,C) is either 0 or 1 for any values of B and C.

$\endgroup$
2
  • 3
    $\begingroup$ Lovely example! $\endgroup$
    – user67724
    Oct 8, 2022 at 18:06
  • 13
    $\begingroup$ In particular, applying this to a 2x2 chessboard results in the example where $A,B,C$ are all uniformly random bits, but correlated so that any pair is independent and the third is their XOR. (The example in guy's answer.) $\endgroup$
    – usul
    Oct 8, 2022 at 20:58
28
$\begingroup$

Imagine $A$ and $B$ are independent Bernoulli random variables with success probability 1/2, and let $C = AB + (1 - A)(1 - B)$ (if you like, $A$ is the event that a coin lands heads, $B$ is the event that another coin lands heads, and $C$ is the event that both coins land the same side up). Then $P(A = 1 \mid C = 1) = 1/2$ but $P(A = 1 \mid B = 1, C = 1) = 1$.

A bit more generally, a common situation where this is not true is when $(A, B, C)$ form a v-structure, i.e., where $A \rightarrow C \leftarrow B$, where $A$ and $B$ are marginally independent but where $B$ can provide information about $A$ if both cause $C$.

$\endgroup$
18
$\begingroup$

Let's continue your example with two decks of cards, supposing that one card is chosen randomly from each deck. You have proposed that $A$ is the event that the ace of spades is chosen from the first deck and that $B$ is the event that the ace of spades is chosen from the second deck.

Now let $C$ be the event that the two cards from the two decks are of different colors. Then $$P(A|C) = P(A) = 1/52$$ but $$P(A|B,C) = 0$$ because $A$, $B$ and $C$ cannot simultaneously be true.

Note that any two of $A$, $B$ and $C$ are independent but the three events are dependent when taken together.

$\endgroup$
2
  • $\begingroup$ That was witty. +1. $\endgroup$ Oct 8, 2022 at 9:01
  • 1
    $\begingroup$ Hmm. The Ace of Spades is always a black card ... $\endgroup$ Oct 8, 2022 at 11:47
15
$\begingroup$

There are two situations how this can happen.

Collider bias

A common way how the situation can occur is collider bias.

An example is in the image below. In this example, intelligence and appearance are statistically unrelated (there's not even an effect in this case as with the coin example below). But conditional on the sum of intelligence and appearance the two become correlated.

example of bias

Invariant effect

With collider bias both A and B have an effect on C. In some special cases, the situation may also appear when B has a causal effect on A, but it does not appear as a statistical dependency.

The underlying reason why this counterintuitive situation can happen is because: When we speak about statistical independence between A and B then it doesn't necessarily mean that A and B have no functional/causal relationship.

Let's say that we flip two fair coins $B$ and $C$, and assign values $-1$ and $1$ to the sides of the coins. Let's define $A := B \times C$.

 B   C   A := B*C
 1   1   1
 1  -1  -1
-1   1  -1
-1  -1   1

Now $B$ does influence the outcome $A$ but $A$ is not statistically dependent on $B$ alone. Namely we have that $P(A = 1) = P(A = -1) = 0.5$ independent of the value of $B$.

Variants of this case occur when $B$ is a variable that influences $A$ while leaving the distribution of $A$ unchanged. So this case happens when $B$ has an effect on $A$ that leaves the distribution of $A$ invariant. E.g. it happens when $B$ switches labels of a fair coin or dice.

This seems like a contrived case, but it is also a bit about a particular interpretation of probability. When we flip a coin, does it matter whether we start with heads up or tails up? Statistically speaking there might not be a relationship. But in the underlying physical process the starting position of the coin has (according to theory) an influence on the position of the coin after the flip. The situation may, after all, be considered deterministic; but we describe it as probabilistic because we have no complete knowledge about the begin state of the entire system and all the parameters that are involved in the outcome.

$\endgroup$
8
  • 2
    $\begingroup$ IIRC, there was another post where you mentioned about the collider bias, the first time I got acquainted with it. +1. $\endgroup$ Oct 8, 2022 at 18:21
  • 1
    $\begingroup$ @User1865345 it might be the post from which I copied the image stats.stackexchange.com/a/579222/164061 $\endgroup$ Oct 8, 2022 at 18:25
  • $\begingroup$ Thanks for the answer! In the collider example, it would help to say with $A$, $B$, and $C$ are. $\endgroup$
    – Dahn
    Oct 9, 2022 at 10:20
  • $\begingroup$ I like the idea of collider bias but I don't agree with your characterization of the second situation. I don't think it is correct to argue that "B does influence A" and "in some sense A is not independent of B" when in fact A and B are unambiguously mathematically independent. I don't accept the apparent implication that "statistical independence" is a flawed concept that misses some important type of dependence. $\endgroup$ Oct 9, 2022 at 22:14
  • $\begingroup$ @GordonSmyth the second situation relates to the idea that 'dependence' can be an ambiguous term. On the one hand you have 'statistical dependence' on the other hand you have 'causal dependence'. In the second example A := B*C is defined as a function of B and by definition dependent on B. All other things equal, B will change the value of A. I am not sure what there is to not accept. The example clearly shows a causal relationship, namely A := XOR(B,C), while A and B can be statistically independent if B and C are Bernoulli variables with p=0.5. $\endgroup$ Oct 9, 2022 at 22:35
5
$\begingroup$

I quote Joseph Blitzstein's Introduction to Probability (2019 2 edn), p. 65.

Example 2.5.11 (Independence doesn't imply conditional independence).

My friends Alice and Bob are the only two people who ever call me on the phone. Each day, they decide independently whether to call me that day. Let A be the event that Alice calls me next Friday and B be the event that Bob calls me next Friday. Assume A and B are unconditionally independent with P(A) > 0 and P(B) > 0.

However, given that I receive exactly one call next Friday, A and B are no longer independent: the call is from Alice if and only if it is not from Bob. In other words, letting C be the event that I receive exactly one call next Friday, $P(B|C) > 0$ while $P(B|A,C) = 0$, so A and B are not conditionally independent given C.

$\endgroup$
4
$\begingroup$

As @Sextus Empiricus (the ancient skeptic!) said, collider bias phenomenon in epidemiology is a great exemplar of why marginal independence of two random variables does not necessarily imply their conditional independence, given a third random variable.

Sackett (1978)'s example is a classic for the collider bias. In his paper, $A =$ having locomotor disease, and $B = $ having respiratory disease. Let $C$ denote the event that a person is hospitalized (for whatever reason). He observed that among hospitalized patients, there was a strong association between $A$ and $B$, whereas there was no association between $A$ and $B$ in the overall population (mix of both $C$ and $\bar C$). If $C$ denotes hospitalization, then we see that $P(A|B,C) \ne P(A|C)$ and $P(B|A,C) \ne P(B|C)$, even though $P(A|B) = P(B)$, this is because both $A$ and $B$ increase the probability of $C$. Hospitalization, $C$, is the collider, being a common effect of $A$ and $B$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.