Pictorial explanation of Statistical Independence

Figures such as a), b) in this drawing are used to explain the idea of (in)dependence between two random variables (named X,Y).

In figure a), knowing the value of X tells us nothing about Y, so X,Y are independent. In figure b), if X is near the right side of its range of data, tracing the "marginal" line upward show restricted range of Y values that are possible, with fewer values as the X value approaches the "corner" of the data.

Ok, that makes sense!

But what about the case shown in Figure c)? The figure show a case where there is a high probability that X,Y occur at one of the four indicated points, with a much lower (but still nonzero) probability of data occuring anywhere in the X-Y square, like in case a). The latter uniform-in-X,Y probabilty is hard to show in the quick sketch, but please imagine it. I attempted to draw it with faint dots.

I believe this case c) also show independent RVs; the joint probability factorizes $$P(X,Y) = P(X)P(Y)$$ But according to the visual logic of the previous explanation, this case must be dependent I think. Looking at the marginal pdf of X, there are two high probability values, and the rest low probability. Knowing that X is at one of the high-probability values means that P(Y|X) has two high-probability values, and any other value is low-probability. But if X is at one of the low-probability values, P(Y|X) is uniform. So knowing X tells us something about Y, $$P(Y|X) != P(Y)$$

Figure d) shows a case with the same marginal probabilities as figure c), but which is not independent (I believe). This may not be needed for our discussion.

Important to clarify, my question is really about the picture and its interpretation, not so much about the math. Apology to C.Rau, who wrote a good answer about the math, but I did not explain the question well enough that it is about the interpretation of the picture not the math.

Specifically, focus on this statement (call it the "visual statement"): if X is near the right side of its range of data, tracing the "marginal" line upward show restricted range of Y values that are possible, with fewer values as the X value approaches the "corner" of the data This statement incorrectly suggests that if the vertical profile at a location $x$ changes as $x$ changes, the two variables are dependent.

I now see that part of the issue is what is meant by considering a vertical profile (showing the density of $Y$ at a particular $X=x$ location): is it normalized to be a probability, or not. Consider a picture of a Gaussian. The vertical (Y) profile clearly changes as a function of $x$, however after normalizing so that the profile represents $p(Y|X=x)$ (a valid probability), then the profile does not change regardless of which $x$ is taken.

The case c) puzzled me. I believed that it showed an independent distribution, but the visual statement does not work for it.

However I see that this picture does not show an independent distribution. If it were independent, the "background" areas would not be constant but instead have to show a smaller version with two ridges corresponding to the spikes. That is, the marginal pdf for $Y$ is

Y=large if X=one of two locations, else small


Then to make an independent plot, at an X location that is not centered on one of the spikes, this Y profile gets multiplied by "small", so it would have the values small*small (mostly) and small*large (at two points). Thus it would not be a constant small value, rather there would be "ridges".

The random variables X,Y in c) are indeed dependent, as per your explanation below the formula "P(X,Y)=P(X)P(Y)", which is just a reformulation of the expression with the conditional probability. First, since P is a probability, it would be more correct to replace the random variable "X" by the event "X in A", where A is some set, and similar "Y" by the event "Y in B". Now, since the event "X in A" has positive probability in the setting that you mention, there is no problem with dividing by it as required in the conditional probability.

Another correction: "Y can take on two possible values" is not correct for c), assuming I understand that picture correctly: the (marginal) distribution of Y has two so-called atoms (one-element sets, called singletons, of positive probability) and all other singletons receive zero probability under that distribution, as is the case with all "fully" continuous distributions (uniform, normal, etc); but Y need not take one of the atoms as its values.

• OK, I made an error in the post, it is corrected now. First the example was 4 non-zero probability points, zero elsewhere, but then I changed to have high/low rather then zero/non-zero, except some of the text was not updated. The corrected sentences begin "Looking at the marginal pdf of X, there are two high probability values, and the rest low probability...." – beginner Mar 19 '18 at 2:59
• With the fix, my original question remains: it seems that the "visual logic" of the picture gives a different answer than the math. – beginner Mar 19 '18 at 3:00
• I did interpret c) in the sense you meant, i.e. high/low rather than zero/nonzero probability at/outside the four distinct dots. I am not sure where the problem is -- note perhaps that the sets A and B that I referred to in the first paragraph are typically intervals; for a continuous random variable (such as normal, uniform), and for a mixture of continuous and uniform (such as your X and Y), you do not get the full information about the distribution by stating the singleton probabilities "P(X=x)". (Apologies if I missed your point.) – Christian Rau Mar 20 '18 at 8:53