7
$\begingroup$

Graphic depicting a Venn Diagram. The sample space, S, has two events in it, A and B, and these overlap. Outcomes x1 and x2 occur in A, outcomes x2 and x3 occur in B and outcome x4 happens outside A and B.

Suppose that we have the situation as depicted in the figure: a random experiment which has 4 outcomes $x_1, ..., x_4$ and two events $A$ and $B$. Also assume that $P(x_i)=0.25$.

Now, since $P(A \cap B) = P(A)P(B) = 0.25$, by definition the events $A$ and $B$ are independent. This does not make sense to me. Why are they independent? Is there an intuitive explanation?

Another thing is, suppose that there is a 5th outcome, $x_5$, outside of $A \cup B$, in that case the events $A$ and $B$ are no longer independent since $P(A \cap B) \ne P(A)P(B)$. This result also does not make sense to me.

$\endgroup$
  • $\begingroup$ Please do not post same question twice! $\endgroup$ – Tim Sep 18 '17 at 19:47
  • $\begingroup$ I thought that my other question was not clear, so I decided to draw a figure and post a new one, hope it does not violate the rules. $\endgroup$ – Sanyo Mn Sep 18 '17 at 19:56
  • 1
    $\begingroup$ It looks like you are rehashing your previous questions, such as stats.stackexchange.com/questions/303011/…. This one is hard to understand because you don't explain how the existence of $x_5$ is supposed to modify the probabilities. If it is assigned a probability of $0$, $A$ and $B$ remain independent--by definition. $\endgroup$ – whuber Sep 18 '17 at 20:19
  • $\begingroup$ @whuber I assume that after $x_5$ is added, all five outcomes will have equal probability, that is, 0.2. The thing is I can do the math and show that in this case $A$ and $B$ are dependent but my problem is how something outside of events $A$ and $B$ can destroy their independence. The addition of $x_5$ should at most decrease the probabilities of $A$ and $B$, it should not effect their independence, says my intuition. $\endgroup$ – Sanyo Mn Sep 18 '17 at 20:53
  • 3
    $\begingroup$ You are attempting to apply an intuition about causal dependence to a concept (independence of events) that is unrelated to that. The best (and only correct) way to develop an intuition about any mathematical definition is to start with the definition and think through its implications, rather than imposing some possibly incorrect preconceptions on it. $\endgroup$ – whuber Sep 19 '17 at 15:37
8
$\begingroup$

Independence means the Venn diagram can be drawn in a simpler way.

After presenting a simple analysis, which is trivial but enlightening, I offer a way of visualizing and generalizing independence and then discuss some of its uses and implications.

Analysis

Two events $A$ and $B$ in the same probability space $\Omega$ determine four events altogether by means of their complements ${A}^\prime = \Omega\setminus A$ and ${B}^\prime = \Omega\setminus B$; namely, the four possible nontrivial intersections $A\cap B$, $A\cap B^\prime$, $A^\prime \cap B$, and $A^\prime \cap B^\prime$. These four events are mutually exclusive--any two have null intersection--and their union is all of $\Omega$.

In general, the probabilities associated with these four intersections could be any values consistent with the axioms: they must be non-negative and sum to unity. (This implies three parameters are needed to describe all such probabilities; the fourth probability is determined by the sum-to-unity constraint.) But when $A$ and $B$ are independent, this simplifies.

Recall that $A$ and $B$ are independent when $\Pr(A\cap B)=\Pr(A)\Pr(B)$. Notice this implies that $A$ and $B^\prime$ are independent, because

$$\eqalign{\Pr(A)&=\Pr(A\cap \Omega) = \Pr(A\cap(B\cup B^\prime))=\Pr((A\cap B)\cup(A\cap B^\prime)) \\&= \Pr(A\cap B)+\Pr(A\cap B^\prime)}$$

implies

$$\eqalign{\Pr(A\cap B^\prime) &= \Pr(A) - \Pr(A\cap B) = \Pr(A) - \Pr(A)\Pr(B) = \Pr(A)\left(1 - \Pr(B)\right) \\&= \Pr(A)\Pr(B^\prime).}$$

Exchanging the roles of $A$ and $B$ in this argument shows $A^\prime$ and $B$ are independent and, finally, replacing $B$ with $B^\prime$ (whence $B^{\prime\prime}=B$) shows $A^\prime$ and $B^\prime$ are independent.

Visualization

This analysis can be depicted by representing $\Omega$ (abstractly) as an interval of points on an axis. $A$ is a subset of this interval and $A^\prime$ is the remainder of the subset. I will make the lengths of these subintervals proportional to their probabilities.

Let's erect another vertical axis, again representing $\Omega$, on which we may draw $B$. We are free to re-order the elements of $\Omega$ on this axis so that $B$ also appears as a subinterval and $B^\prime$ is the remainder, again drawn with lengths proportional to their probabilities.

Figure

These intervals determine rectangles in the figure, as shown. Independence of $A$ and $B$ means the relative areas of the rectangles are their probabilities.

Discussion

Now only two parameters, instead of three, are needed to describe all possible probability distributions: $\Pr(A)$ and $\Pr(B)$ completely determine all the rectangle areas.

This idea generalizes. Let $A_1, A_2, \ldots, A_m$ be events that partition $\Omega$: that is, the intersection of any distinct pair of them is empty and their union is $\Omega$. Let $B_1, B_2, \ldots, B_n$ be another partition. These two partitions are independent when $\Pr(A_i\cap B_j) = \Pr(A_i)\Pr(B_j)$ for all $i,j$. We may draw a similar figure in which the $A_i$ are a sequence of non-overlapping line segments on the x axis and the $B_j$ are a sequence of non-overlapping line segments on the y axis, each with a length proportional to the probability. This generalized idea of independence simply means the probabilities of all $m\times n$ rectangles formed by these seqments are determined by the $m$ probabilities for the $A_i$ and the $n$ probabilities for the $B_j$. That replaces $mn$ numbers (subject to a single sum-to-unity constraint) by $m+n$ numbers (subject to two separate sum-to-unity constraints). The reduction in parameter counts from $mn-1$ to $m+n-2$ quantifies how much simplification has occurred. It's substantial.

This kind of diagram can help your intuition in various ways. When you think of independence, think of two one-dimensional axes filling out a two-dimensional region and think of areas of rectangles determined by the lengths of their sides. If you progress in your study of probability far enough theoretically, eventually you will encounter generalizations in which the concept of independence extends to "sub sigma algebras." (A sub sigma algebra is a collection of events having some additional properties that don't matter here. It's a way to generalize the finite partitions, as previously described, into infinite partitions.) If you visualize a "sub sigma algebra" as a collection of intervals on a line (although this time they may overlap each other), you will not need to enlarge or modify your intuition one bit: this hugely general and abstract definition of independence merely says that any rectangle formed by a set on the x-axis and a set on the y-axis has a probability proportional to its area.

Yet another generalization extends to independence of three or more sets (or sub sigma algebras). Visualize these by adding more axes to the picture: a third axis in a third dimension for the third set (now the relevant probabilities are volumes of cuboids), and so on. In effect, independence lets us break down a potentially complicated probability space into simpler "one-dimensional" components, almost in the same way we analyze vectors (in vector spaces) in terms of their components.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

There are a few ways to show that two events are independent, but I'll work through one of them to address your two scenarios.

Scenario 1 - The Figure As you point out, events $A$ and $B$ are independent in the situation you describe. This is because:

$$P(A \cap B) = P(A)P(B) \\ P(x_2) = [P(x_1)+P(x_2)] \times [P(x_2)+P(x_3)]\\ 0.25 = [0.25+0.25] \times [0.25+0.25] \\ 0.25 = 0.25$$

But why does the fact that $P(A \cap B) = P(A)P(B)$ mean that events $A$ and $B$ are independent? We can show this using Bayes's Rule:

$$P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{P(A)P(B)}{P(B)} = P(A)$$

Or, in English, the probability that event $A$ occurs given that event $B$ occurs is equal to the probability of event $A$ occurring with no knowledge regarding $B$. This means that event $B$ provides us with no information about $A$ (the opposite can be shown the exact same way) and this means that the two events occur independently of each other.

Scenario 2 - Outcome $x_5$ You haven't given us quite enough information to work through this fully, but if we assume that adding $x_5$ as you described does not affect $P(A)$ or $P(B)$ then $A$ and $B$ are still independent by the calculation shown above.

If we assume that $P(x_i) = 0.2$ now, then we can show:

$$P(A | B) = \frac{P(A \cap B)}{P(B)} \\ P(A | B) = \frac{P(x_2)}{P(x_2)+P(x_3)} \\ P(A | B) = \frac{0.2}{0.4} \\ P(A | B) = 0.5 \ne P(A) = 0.4$$

The intuitive explanation of these numbers is that in this new scenario, whether or not $B$ has occurred affects the likelihood that $A$ has occurred:

  • Without any information about the outcome, we know that $A$ has a $40\%$ chance of occurring;
  • if we are told that $B$ has occurred, however, then we know that $A$ has a $50\%$ chance of occurring; and
  • if we are told that $B$ has NOT occurred, then we know that $A$ has a $33\%$ chance of occurring.

Event $A$ is more likely or less likely to occur based on whether or not event $B$ has occurred, which means that the events are not independent and, equivalently, that they are dependent.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ but why the addition of $x_5$ changes the independence of events $A$ and $B$ $\endgroup$ – Sanyo Mn Sep 18 '17 at 19:52
  • $\begingroup$ thank you for your answer @user77876, actually, I can make these calculations, but the thing is, it still does not make sense to me, and I don't know how to further explain my problem. $\endgroup$ – Sanyo Mn Sep 18 '17 at 19:58
  • $\begingroup$ cup should be a cap directly after first derivation $\endgroup$ – Gijs Sep 18 '17 at 20:10
  • 1
    $\begingroup$ @Gijs Thanks for the tip. Those two are too easy to typo. $\endgroup$ – user77876 Sep 18 '17 at 21:52
  • $\begingroup$ @SanyoMn I've updated my explanation for the second scenario to try and put it "intuitively." The summary is that, as shown by the calculations, in the second scenario "Event A is more or less likely to occur based on whether or not event B has occurred, which means that the events are not independent and, equivalently, that they are dependent." $\endgroup$ – user77876 Sep 18 '17 at 22:53
2
$\begingroup$

The intuition of independence is clearer if you think about conditional probability. Let us define the conditional probability $P(B \mid A) := P(A \cap B) / P(A)$; intuitively, this is the probability that $B$ is true given that you know $A$ is true. In terms of the Venn diagram, this is the proportion of $A$ that the intersection $A \cap B$ takes up.

Then, independence would imply $$P(B \mid A) = \frac{P(A \cap B)}{P(A)} = \frac{P(A) P(B)}{P(A)} = P(B),$$ which in English means that the probability of $B$ did not "change" even after gaining the extra knowledge that $A$ is true. This is the sense of "independence."

In terms of actual numbers in your example, we have $P(B) = 1/2$, but also $P(B \mid A) = \frac{1/4}{1/2} = \frac{1}{2}$, so gaining the knowledge that $A$ is true did not "affect" the probability of $B$, hence "independence."


Edit: the reason why, if you add $x_5$ and again assuming uniformity over the atoms ($P(x_i) = 0.2$ for all $i$), that independence breaks is that $P(B) = 2/5$, but if you now know the $A$ is true, then the probability of $B$ being true is $P(B \mid A) = P(A \cap B) / P(A) = \frac{1/5}{2/5} = 1/2$, because the proportion of $A$ that is occupied by the event $B$ is larger than the proportion of the entire sample space that is occupied by the event $B$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ but why the addition of $x_5$ changes the independence of events $A$ and $B$ $\endgroup$ – Sanyo Mn Sep 18 '17 at 19:52
1
$\begingroup$

Let's say that you have $n$ elements in your finite sample space, $x_1, x_2, \cdots x_n$, each with equal probability $P(x_i)=1/n$.

Let $i, j, k$ all be different and between $1$ and $n$. For the ease of notation I write $x_i$ in stead of $\{x_i\}$ in the formulas below.

Then $P(x_i \cup x_j)=2/n$ while $P(x_i \cup x_j|_{x_j \cup x_k})=0.5$.

So both are equal iff $n=4$. This means that, if $n>4$ then the fact that you already observed $x_j \cup x_k$ gives you additional information on the occurrence of $x_i \cup x_j$ because knowing that you already observed $x_j \cup x_k$ makes it more probable ($P(x_i \cup x_j|_{x_j \cup x_k})=0.5$) to observe $x_i \cup x_j$ compared to lacking the knowledge that $x_j \cup x_k$ was observed ($P(x_i \cup x_j)=2/n<0.4$ when $n>4$).

If $n=4$ then the fact that you observed $x_j \cup x_k$ gives you no additional observation on the probability of observing $x_i \cup x_j$.

So the fact that your sample space is smaller (and it only holds for $n=4$ in this setting) creates cases where observing something does not bring knowledge on the (probability of) observing other things but that may change when you have larger (n>4) sample spaces.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Could you explain what you are trying to denote by $P(x_i \cup x_j |_{x_j \cup x_k} )$? I don't quite understand what you are trying to say with that, is it the probability of $x_i$ or $x_j$ given that $x_j$ or $x_k$ occurred (Which would explain the $\frac 12$ pretty obviously) ? Or is it something else, like the probabliity of $x_i$ or $x_j$ given that $x_j$ is either $x_j$ or $x_K$ (this one makes even less sense to me than the first). I just don't clearly understand what your intending to say in that line. $\endgroup$ – Shinaolord Oct 3 '19 at 15:04
1
$\begingroup$

Let's go through your two coins example provided in the comment. The data presented in the table looks like below.

$$ \begin{array}{ccccc} X_1 & X_2 & & & \text{Prob} \\ t & t & & B & 1/4\\ t & h & & & 1/4\\ h & t & A & & 1/4\\ h & h & A & B & 1/4 \end{array} $$

You can observe that $P(A) = 0.5$ and $P(B) = 0.5$. The events are independent since $P(A \cap B) = P(A)\,P(B) = 0.25$. You say that the result is "unintuitive" because

... knowing that $B$ has occurred eliminates the possibility of a HT outcome which is one of the outcomes in event $A$. So, intuitively occurrence of $B$ has an effect on $A$ ...

But look at the table once again. If $B$ has occurred (rows 1 & 4), then only in the half of the cases $A$ occurs (row 4), so $P(A) = P(A|B) = 0.5$. Same if $A$ has occurred (rows 3 & 4), then $P(B) = P(B|A) = 0.5$. They are independent. So knowing that one of the events occurred tells you nothing about the other. This makes perfect sense.

Regarding your comment,

... For example I don't agree with your sentence "knowing that one of the events occurred tells you nothing about the other". Because knowing $B$ tells me something about $A$, namely, the HT outcome in $A$ cannot occur.

I guess that this is where your confusion has it's roots. The event $A$ is defined as $X_1 = h$, so event $A$ is observed, or not, no matter what is $X_2$. For $A$ to happen it doesn't matter that $ht$ cannot occur. Obviously the event $X_1 = h \land X_2 = t$ depends on $B$ since $P(X_1 = h \land X_2 = t) = 0.25$ while $P(X_1 = h \land X_2 = t \mid B) = 0$, but the event $A$ itself does not.

Imagine that the two coin tosses happen behind a curtain and you don't see the outcomes of the tosses, but you are only informed about occurrence of $A$ or $B$. Information about happening of one of the events does not help you to guess if the second one has happened.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ my problem I think kind of linguistic. For example I don't agree with your sentence "knowing that one of the events occurred tells you nothing about the other". Because knowing $B$ tells me something about $A$, namely, the HT outcome in $A$ cannot occur. But knowing $B$ does not change the probability of $A$. On the other hand, knowing the outcome of the first coin toss tells me nothing about the outcome of the second toss, I agree with this usage in this case, but there seems to be a difference with this situation and the events $A$ and $B$ the defined above. $\endgroup$ – Sanyo Mn Sep 19 '17 at 11:20
  • $\begingroup$ @SanyoMn see my edit. $\endgroup$ – Tim Sep 19 '17 at 11:36
0
$\begingroup$

Just change the probabilities, say $$P_1=1/2\\P_2=1/4\\P_3=1/8$$, now $$P_A=3/4\\P_B=3/8$$ Hence, $$P(A\times B)=1/4\\P(A)\times P(B)=9/32$$ and $$P(A\times B)\ne P(A)\times P(B)$$

So, you constructed (albeit accidentally) a case where it just happens so that $P(A\times B)= P(A)\times P(B)$ for a special set of probabilities $P_i$. Generally, for another set of probabilities $P_i$ this will not hold.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I believe the idea of independence is, using your Venn diagram, that the proportion of A in B is equal to the proportion of A in Ω. That is, AB takes up half the space of B, exactly as much as A takes out of Ω.

If we now interpret the Venn diagram in terms of probabilities, this means that given that B has occurred, A has ½ of occurring (=0.25/0.5), which is exactly the probability of A occurring anyways (=0.5/1), therefore knowledge/occurrence of B does not change the probability of A.

If you add a fifth event assigning now each outcome a probability of 0.2, then those proportions are not equal anymore.

We can generalize from your illustrative example and this geometric intuition to define independence as equal proportions, P(A|B) = P(A), that is, the probability of A happening (in general) equals the the probability of it happening (specifically) when B has happened, just as in your Venn diagram. Now P(A|B) = P(AB) / P(B), which is the proportion that A occupies in B. Substituting P(A) for P(A|B), we get the formal definition of independence for two events: P(AB) = P(A)P(B)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.