Need intuition about independence of events

Question

Suppose that we have the situation as depicted in the figure: a random experiment which has 4 outcomes $x_1, ..., x_4$ and two events $A$ and $B$. Also assume that $P(x_i)=0.25$.

Now, since $P(A \cap B) = P(A)P(B) = 0.25$, by definition the events $A$ and $B$ are independent. This does not make sense to me. Why are they independent? Is there an intuitive explanation?

Another thing is, suppose that there is a 5th outcome, $x_5$, outside of $A \cup B$, in that case the events $A$ and $B$ are no longer independent since $P(A \cap B) \ne P(A)P(B)$. This result also does not make sense to me.

Answer 1

Independence means the Venn diagram can be drawn in a simpler way.

After presenting a simple analysis, which is trivial but enlightening, I offer a way of visualizing and generalizing independence and then discuss some of its uses and implications.

Analysis

Two events $A$ and $B$ in the same probability space $\Omega$ determine four events altogether by means of their complements ${A}^\prime = \Omega\setminus A$ and ${B}^\prime = \Omega\setminus B$; namely, the four possible nontrivial intersections $A\cap B$, $A\cap B^\prime$, $A^\prime \cap B$, and $A^\prime \cap B^\prime$. These four events are mutually exclusive--any two have null intersection--and their union is all of $\Omega$.

In general, the probabilities associated with these four intersections could be any values consistent with the axioms: they must be non-negative and sum to unity. (This implies three parameters are needed to describe all such probabilities; the fourth probability is determined by the sum-to-unity constraint.) But when $A$ and $B$ are independent, this simplifies.

Recall that $A$ and $B$ are independent when $\Pr(A\cap B)=\Pr(A)\Pr(B)$. Notice this implies that $A$ and $B^\prime$ are independent, because

$$\eqalign{\Pr(A)&=\Pr(A\cap \Omega) = \Pr(A\cap(B\cup B^\prime))=\Pr((A\cap B)\cup(A\cap B^\prime)) \\&= \Pr(A\cap B)+\Pr(A\cap B^\prime)}$$

implies

$$\eqalign{\Pr(A\cap B^\prime) &= \Pr(A) - \Pr(A\cap B) = \Pr(A) - \Pr(A)\Pr(B) = \Pr(A)\left(1 - \Pr(B)\right) \\&= \Pr(A)\Pr(B^\prime).}$$

Exchanging the roles of $A$ and $B$ in this argument shows $A^\prime$ and $B$ are independent and, finally, replacing $B$ with $B^\prime$ (whence $B^{\prime\prime}=B$) shows $A^\prime$ and $B^\prime$ are independent.

Visualization

This analysis can be depicted by representing $\Omega$ (abstractly) as an interval of points on an axis. $A$ is a subset of this interval and $A^\prime$ is the remainder of the subset. I will make the lengths of these subintervals proportional to their probabilities.

Let's erect another vertical axis, again representing $\Omega$, on which we may draw $B$. We are free to re-order the elements of $\Omega$ on this axis so that $B$ also appears as a subinterval and $B^\prime$ is the remainder, again drawn with lengths proportional to their probabilities.

These intervals determine rectangles in the figure, as shown. Independence of $A$ and $B$ means the relative areas of the rectangles are their probabilities.

Discussion

Now only two parameters, instead of three, are needed to describe all possible probability distributions: $\Pr(A)$ and $\Pr(B)$ completely determine all the rectangle areas.

This idea generalizes. Let $A_1, A_2, \ldots, A_m$ be events that partition $\Omega$: that is, the intersection of any distinct pair of them is empty and their union is $\Omega$. Let $B_1, B_2, \ldots, B_n$ be another partition. These two partitions are independent when $\Pr(A_i\cap B_j) = \Pr(A_i)\Pr(B_j)$ for all $i,j$. We may draw a similar figure in which the $A_i$ are a sequence of non-overlapping line segments on the x axis and the $B_j$ are a sequence of non-overlapping line segments on the y axis, each with a length proportional to the probability. This generalized idea of independence simply means the probabilities of all $m\times n$ rectangles formed by these seqments are determined by the $m$ probabilities for the $A_i$ and the $n$ probabilities for the $B_j$. That replaces $mn$ numbers (subject to a single sum-to-unity constraint) by $m+n$ numbers (subject to two separate sum-to-unity constraints). The reduction in parameter counts from $mn-1$ to $m+n-2$ quantifies how much simplification has occurred. It's substantial.

This kind of diagram can help your intuition in various ways. When you think of independence, think of two one-dimensional axes filling out a two-dimensional region and think of areas of rectangles determined by the lengths of their sides. If you progress in your study of probability far enough theoretically, eventually you will encounter generalizations in which the concept of independence extends to "sub sigma algebras." (A sub sigma algebra is a collection of events having some additional properties that don't matter here. It's a way to generalize the finite partitions, as previously described, into infinite partitions.) If you visualize a "sub sigma algebra" as a collection of intervals on a line (although this time they may overlap each other), you will not need to enlarge or modify your intuition one bit: this hugely general and abstract definition of independence merely says that any rectangle formed by a set on the x-axis and a set on the y-axis has a probability proportional to its area.

Yet another generalization extends to independence of three or more sets (or sub sigma algebras). Visualize these by adding more axes to the picture: a third axis in a third dimension for the third set (now the relevant probabilities are volumes of cuboids), and so on. In effect, independence lets us break down a potentially complicated probability space into simpler "one-dimensional" components, almost in the same way we analyze vectors (in vector spaces) in terms of their components.

Answer 2

There are a few ways to show that two events are independent, but I'll work through one of them to address your two scenarios.

Scenario 1 - The Figure As you point out, events $A$ and $B$ are independent in the situation you describe. This is because:

$$P(A \cap B) = P(A)P(B) \\ P(x_2) = [P(x_1)+P(x_2)] \times [P(x_2)+P(x_3)]\\ 0.25 = [0.25+0.25] \times [0.25+0.25] \\ 0.25 = 0.25$$

But why does the fact that $P(A \cap B) = P(A)P(B)$ mean that events $A$ and $B$ are independent? We can show this using Bayes's Rule:

$$P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{P(A)P(B)}{P(B)} = P(A)$$

Or, in English, the probability that event $A$ occurs given that event $B$ occurs is equal to the probability of event $A$ occurring with no knowledge regarding $B$. This means that event $B$ provides us with no information about $A$ (the opposite can be shown the exact same way) and this means that the two events occur independently of each other.

Scenario 2 - Outcome $x_5$ You haven't given us quite enough information to work through this fully, but if we assume that adding $x_5$ as you described does not affect $P(A)$ or $P(B)$ then $A$ and $B$ are still independent by the calculation shown above.

If we assume that $P(x_i) = 0.2$ now, then we can show:

$$P(A | B) = \frac{P(A \cap B)}{P(B)} \\ P(A | B) = \frac{P(x_2)}{P(x_2)+P(x_3)} \\ P(A | B) = \frac{0.2}{0.4} \\ P(A | B) = 0.5 \ne P(A) = 0.4$$

The intuitive explanation of these numbers is that in this new scenario, whether or not $B$ has occurred affects the likelihood that $A$ has occurred:

• Without any information about the outcome, we know that $A$ has a $40\%$ chance of occurring;
• if we are told that $B$ has occurred, however, then we know that $A$ has a $50\%$ chance of occurring; and
• if we are told that $B$ has NOT occurred, then we know that $A$ has a $33\%$ chance of occurring.

Event $A$ is more likely or less likely to occur based on whether or not event $B$ has occurred, which means that the events are not independent and, equivalently, that they are dependent.

Answer 3

The intuition of independence is clearer if you think about conditional probability. Let us define the conditional probability $P(B \mid A) := P(A \cap B) / P(A)$; intuitively, this is the probability that $B$ is true given that you know $A$ is true. In terms of the Venn diagram, this is the proportion of $A$ that the intersection $A \cap B$ takes up.

Then, independence would imply $$P(B \mid A) = \frac{P(A \cap B)}{P(A)} = \frac{P(A) P(B)}{P(A)} = P(B),$$ which in English means that the probability of $B$ did not "change" even after gaining the extra knowledge that $A$ is true. This is the sense of "independence."

In terms of actual numbers in your example, we have $P(B) = 1/2$, but also $P(B \mid A) = \frac{1/4}{1/2} = \frac{1}{2}$, so gaining the knowledge that $A$ is true did not "affect" the probability of $B$, hence "independence."

---

Edit: the reason why, if you add $x_5$ and again assuming uniformity over the atoms ($P(x_i) = 0.2$ for all $i$), that independence breaks is that $P(B) = 2/5$, but if you now know the $A$ is true, then the probability of $B$ being true is $P(B \mid A) = P(A \cap B) / P(A) = \frac{1/5}{2/5} = 1/2$, because the proportion of $A$ that is occupied by the event $B$ is larger than the proportion of the entire sample space that is occupied by the event $B$.

Answer 4

Let's say that you have $n$ elements in your finite sample space, $x_1, x_2, \cdots x_n$, each with equal probability $P(x_i)=1/n$.

Let $i, j, k$ all be different and between $1$ and $n$. For the ease of notation I write $x_i$ in stead of $\{x_i\}$ in the formulas below.

Then $P(x_i \cup x_j)=2/n$ while $P(x_i \cup x_j|_{x_j \cup x_k})=0.5$.

So both are equal iff $n=4$. This means that, if $n>4$ then the fact that you already observed $x_j \cup x_k$ gives you additional information on the occurrence of $x_i \cup x_j$ because knowing that you already observed $x_j \cup x_k$ makes it more probable ($P(x_i \cup x_j|_{x_j \cup x_k})=0.5$) to observe $x_i \cup x_j$ compared to lacking the knowledge that $x_j \cup x_k$ was observed ($P(x_i \cup x_j)=2/n<0.4$ when $n>4$).

If $n=4$ then the fact that you observed $x_j \cup x_k$ gives you no additional observation on the probability of observing $x_i \cup x_j$.

So the fact that your sample space is smaller (and it only holds for $n=4$ in this setting) creates cases where observing something does not bring knowledge on the (probability of) observing other things but that may change when you have larger (n>4) sample spaces.

Answer 5

Let's go through your two coins example provided in the comment. The data presented in the table looks like below.

$$ \begin{array}{ccccc} X_1 & X_2 & & & \text{Prob} \\ t & t & & B & 1/4\\ t & h & & & 1/4\\ h & t & A & & 1/4\\ h & h & A & B & 1/4 \end{array} $$

You can observe that $P(A) = 0.5$ and $P(B) = 0.5$. The events are independent since $P(A \cap B) = P(A)\,P(B) = 0.25$. You say that the result is "unintuitive" because

... knowing that $B$ has occurred eliminates the possibility of a HT outcome which is one of the outcomes in event $A$. So, intuitively occurrence of $B$ has an effect on $A$ ...

But look at the table once again. If $B$ has occurred (rows 1 & 4), then only in the half of the cases $A$ occurs (row 4), so $P(A) = P(A|B) = 0.5$. Same if $A$ has occurred (rows 3 & 4), then $P(B) = P(B|A) = 0.5$. They are independent. So knowing that one of the events occurred tells you nothing about the other. This makes perfect sense.

Regarding your comment,

... For example I don't agree with your sentence "knowing that one of the events occurred tells you nothing about the other". Because knowing $B$ tells me something about $A$, namely, the HT outcome in $A$ cannot occur.

I guess that this is where your confusion has it's roots. The event $A$ is defined as $X_1 = h$, so event $A$ is observed, or not, no matter what is $X_2$. For $A$ to happen it doesn't matter that $ht$ cannot occur. Obviously the event $X_1 = h \land X_2 = t$ depends on $B$ since $P(X_1 = h \land X_2 = t) = 0.25$ while $P(X_1 = h \land X_2 = t \mid B) = 0$, but the event $A$ itself does not.

Imagine that the two coin tosses happen behind a curtain and you don't see the outcomes of the tosses, but you are only informed about occurrence of $A$ or $B$. Information about happening of one of the events does not help you to guess if the second one has happened.

Answer 6

Just change the probabilities, say $$P_1=1/2\\P_2=1/4\\P_3=1/8$$, now $$P_A=3/4\\P_B=3/8$$ Hence, $$P(A\times B)=1/4\\P(A)\times P(B)=9/32$$ and $$P(A\times B)\ne P(A)\times P(B)$$

So, you constructed (albeit accidentally) a case where it just happens so that $P(A\times B)= P(A)\times P(B)$ for a special set of probabilities $P_i$. Generally, for another set of probabilities $P_i$ this will not hold.

Answer 7

I believe the idea of independence is, using your Venn diagram, that the proportion of A in B is equal to the proportion of A in Ω. That is, AB takes up half the space of B, exactly as much as A takes out of Ω.

If we now interpret the Venn diagram in terms of probabilities, this means that given that B has occurred, A has ½ of occurring (=0.25/0.5), which is exactly the probability of A occurring anyways (=0.5/1), therefore knowledge/occurrence of B does not change the probability of A.

If you add a fifth event assigning now each outcome a probability of 0.2, then those proportions are not equal anymore.

We can generalize from your illustrative example and this geometric intuition to define independence as equal proportions, P(A|B) = P(A), that is, the probability of A happening (in general) equals the the probability of it happening (specifically) when B has happened, just as in your Venn diagram. Now P(A|B) = P(AB) / P(B), which is the proportion that A occupies in B. Substituting P(A) for P(A|B), we get the formal definition of independence for two events: P(AB) = P(A)P(B)