Seeking an intuitive understanding of independence for random variables

Question

I'm trying to gain a deeper, more intuitive understanding of what it means for two random variables $X$ and $Y$ to be independent. From my statistics courses, I know that $X$ and $Y$ are independent iff their joint pdf is the product of their marginals, i.e. $$f_{X,Y}(x,y) = f_X(x)f_Y(y).$$ Moreover, I have learned that independence means that knowing information about one variable gives no additional information about the other. Here's my question: would it be accurate to say that two random variables are independent if they are generated from completely separate processes?

(For context on where I'm coming from, I am taking a course called "Stochastic Processes" and we are learning about algorithms used to generate different kinds of random variables.)

For example: If I use R (or any other language) to generate two random variables, say a normal and a beta using the functions rnorm() and rbeta(), would these two random variables be independent, seeing as the processes which generated them were completely separate/had no influence on each other?

And would it be accurate to say that non-independent random variables are generated from the same or related processes? My stats textbook gives the example of a person's height $X$ and weight $Y$ being non-independent. This of course makes sense; if we know that someone is extremely tall, it's likely they also weigh a substantial amount. In this example, $X$ and $Y$ are clearly related to some degree because they are both attributes of an individual, who "generates" both $X$ and $Y$. Looking more deeply into this example, we know that biologically, the processes by which a person grows taller and the processes by which a person gains weight have overlap. Hence, we expect $X$ and $Y$ to be non-independent.

I hope what I'm trying to say makes sense. Is my intuitive understanding of independent and non-independent random variables correct? I would appreciate any other perspectives that might help me to understand independence more deeply and intuitively.

Answer 1

Welcome to this community!

The following answer comes from the point of view of probability as a generalization of logic, and as an expression of a person's degrees of belief.

First of all I'd like to emphasize that independence is not a physical property of objects or quantities. It is a property of a person's degree of belief about those quantities. It depends on a person's knowledge about those quantities. So two quantities may have independent probabilities for one person, with one state of knowledge, and non-independent probabilities for another person, with a different state of knowledge.

Here's a simple example. Consider a specific individual. Thanks to your background information $I$, you know their height $X$, let's say 170 cm (within 1 cm), and their weight, let's say 70 kg. So for you $$ \begin{aligned} \mathrm{P}(X\!=\!x | I) &= \delta(x, \text{170 cm}), \\ \mathrm{P}(Y\!=\!y | I) &= \delta(y, \text{70 kg}). \end{aligned} $$

You can easily verify that these probabilities are independent: $$ \begin{aligned} \mathrm{P}(X\!=\!x, Y\!=\!y | I) &= \mathrm{P}(X\!=\!x| I)\times \mathrm{P}(Y\!=\!y | I)\\ &=\delta(x,\text{170 cm}) \times \delta(y,\text{70 kg}) \end{aligned} $$

I don't have any information about this person instead (and don't know the information you have either), and I assign a joint probability to $X$ and $Y$ that doesn't factorize, because my background knowledge tells me that information about one quantity can improve my information about the other. In your case this doesn't happen because your information is complete. Note that the same would happen if you had complete information about one quantity but not about the other.

This example may appear a little trivial, but I really invite you to think about it.

Another important point is that independence may hold for the probabilities of specific values of two quantities, but not all their values. Here's an example. Suppose $X\in\{1,2,3\}$ and $Y\in\{1,2\}$. Consider the following joint distribution, written in matrix form: $$ \mathrm{P}(X\!=\!x, Y\!=\!y) = \begin{matrix} &{\scriptstyle y}\\ {\scriptstyle x}\!\!\!\!\mbox{}& \begin{pmatrix} \frac{1}{2} \left(-\frac{\sqrt{41}}{10}-\frac{1}{2}\right)+\frac{3}{5} & \frac{1}{10} \\ \frac{1}{10} & \frac{1}{2} \left(\frac{\sqrt{41}}{10}+\frac{1}{2}\right) \\ \frac{1}{10} & \frac{1}{10} \end{pmatrix} \end{matrix} \approx \begin{pmatrix} 0.0298438 & 0.1 \\ 0.1 & 0.570156 \\ 0.1 & 0.1 \end{pmatrix}. $$ If you do the calculations of the marginal probabilities you'll find that $$\mathrm{P}(X\!=\!1, Y\!=\!1) =\mathrm{P}(X\!=\!1) \times \mathrm{P}(Y\!=\!1) $$ but, for example, $$\mathrm{P}(X\!=\!2, Y\!=\!1) = 0.10 \ne \mathrm{P}(X\!=\!2) \times \mathrm{P}(Y\!=\!1) \approx 0.15. $$ So the probabilities for the specific values $X=1$, $Y=1$ are independent, but those for the specific values $X=2$, $Y=1$ are not. So, again, independence is not the property of two quantities, but of our probabilities for specific values of those quantities.

There may be a physical interdependence between two quantities, in the sense that they may be related by a physical law. But this is different from informational dependence. Often our knowledge about physical dependence leads us to have non-independent beliefs, but not always. Also, the informational dependence of two quantities may be motivated not by a mutual physical dependence, but because we know both are physically influenced by a third quantity.

There's an important fact about independent vs non-independent joint probabilities: if you collect new data or information, a non-independent joint probability can be updated (via Bayes's theorem) to an independent one. But an independent joint probability can never be updated to a non-independent one. So, independent joint probabilities are "irreversible", so to speak.

Regarding the connection with information theory, it's possible to show that the joint probability of two quantities has vanishing mutual information if and only if it is factorizable, that is, independent.

Independence can also be seen as the byproduct of informational irrelevance, that is, the fact that the probability for some value of one quantity is the same whether you know that another quantity has some specific value or not: $$\mathrm{P}(X\!=\!x | Y\!=\!y) = \mathrm{P}(X\!=\!x).$$

There's a good paper by A. P. Dawid on this: Conditional Independence in Statistical Theory. And Jaynes's book Probability Theory: The Logic of Science has many insightful discussions and examples about logical and causal independence and information theory. See for example §§ 4.2–4.3, 6.11–6.12, 10.10, about the distinction between logical independence and causal independence, and chap. 11 about the connection with information theory.