# How do we understand the relationship between independent probabilities and real-world independence?

From what I have come to understand, the events A and B are considered independent for purposes of probability theory when

$$p(A \cap B) = p(A) \cdot p(B)$$

Now, supposing I flip two coins. I write down the probabilities for the joint outcomes as $\left(\frac{1}{4}, \frac{1}{4}, \frac{1}{4}, \frac{1}{4}\right)$ and see that the individual coins are independent. But how do we know those probabilities? Well, the first coin could come up either heads or tails with probability $\frac{1}{2}$, then in the case of heads we know that the probabilities for the second coin are still $\frac{1}{2}$… but this seems to be assuming $p(A|B) = p(A)$, which is mathematically just a rearrangement of the above formula, so this feels like circular reasoning.

So, what's the explanation? Why does theoretical independence correspond to practical independence, in a not-apparently-circular way?

• Real world independence?. When occurrence and non occurrence of a particular object does not depend on occurrence and non occurrence of another object is called the independence of these objects. In real world where this definition stand statistical independence and real world independence are same.
– SAAN
Aug 30, 2013 at 8:56
• By 'real world independence' do you mean something like 'physically independent' or something else? Independence in the probability sense is a jargon term with a very specific meaning which applies to probability models -- mathematical constructs which needn't relate to anything in the real world at all. Aug 30, 2013 at 9:59
• You've assumed independence when constructing the joint distribution. Aug 30, 2013 at 10:06
• In the real world the impact of the lack of complete independence (things like the butterfly effect) is probably less substantial than the lack of precise fairness, i.e. the probability of heads is almost certainly not exactly 1/2. Aug 30, 2013 at 10:20
• Would it be too far-fetched to suppose that people develop theories and concepts to describe what they observe, and therefore what they observe actually does tend sometimes to conform to those theories? :-)
– whuber
Aug 31, 2013 at 2:50

I think if you approach it the other way around it is more intuitively understandable. What does it mean that two variables $A$ and $B$ are independent? It means that knowing the probability distribution/function of one tells us absolutely nothing about the other one. The fact that $B$ may or may not have occurred is irrelevant, extraneous, and pretty much distracting when we are thinking of $A$. What does this mean in probabilistic terms? It means that the probability of $A$ occurring, given that $B$ occurred is the same as the probability of $A$ ignoring $B$ completely. We translate this English (or whatever language you wish) concept into symbolic mathematical form as $P(A|B) = P(A)$. This is the root expression, not the product rule, as this is the translation into mathematical symbols of the meaning of the word "independent".

Now that we have defined/translated "independence" to mean $P(A|B) = P(A)$, we can make the following observation: $$P(A|B) = P(A)\\ \frac{P(A \cap B)}{P(B)} = P(A)\\ P(A \cap B) = P(A)P(B)$$ The "product rule" is an outgrowth (via algebraic manipulation) of the definition, not the other way around.

• What you say makes sense intuitively and is a really nice way of explaning it, but I quibble on "not the other way round". In fact I have seen independence defined via $P(A\cap B)=P(A)P(B)$, and for pdfs, $f_{XY}(x,y)=f_X(x)f_Y(y)$. In the discrete case, starting from (1) $P(A\cap B) = P(A)P(B)$, and the definition of conditional probability (2) $P(A|B)=P(A\cap B)/P(B)$, you can then substitute (1) into (2) to get $P(A|B) = P(A)$. Aug 30, 2013 at 16:48
• I've seen it that way too, but I harken back to my bible on Probability, DeGroot 2ed, and where he introduces the concept of independence (ch. 1.11), the explanations are of the form (paraphrased p. 44) since X and Y are unrelated, the sample space is the product of the individual outcomes *each equally possible*… Which is just the way to say in English $P(A|B) = P(A)$, so I think that even if the independence is defined as the product rule, it is always justified (sometimes only implicitly) using the conditional definition. (Larson 1982 pp.70-71) defines it explicitly using conditionals. Aug 30, 2013 at 17:01
• I like your way of explaining it, but I do not think that this answers my question. Why is the translation in question (from colloquial "provides no information" to the equation) a faithful one? That is really what I am trying to figure out.
– Owen
Aug 30, 2013 at 23:50
• Your explanation is nice but it runs into difficulties when $P(B)=0$. The definition and concept of conditional independence are complicated and difficult in this case, whereas a formula like $P(A\cap B) =P(A)P(B)$ is simple and clear.
– whuber
Aug 31, 2013 at 2:48

I think there's a problem where you say you "observe the two coins are independent." You never observe probabilistic independence, per se; it is always a property of the events / random variables under consideration, which are constructed by the modeller to represent some (physical or otherwise) phenomenon.

So, the physical phenomenon here is: there are two fair coins, and tossing either one of them does not affect the outcome of the other. They're 'independent' in a nontechnical sense, if you want to use that word.

A probabilistic model that we can build to represent this physical phenomenon mathematically is: define two independent, Bernoulli(1/2)-distributed random variables $X_{0}$ and $X_{1}$. I.e.,

$$X_{0}, X_{1} \overset{iid}\sim \text{bernoulli}(1/2) \\$$

Here I mean 'independent' in the technical, probabilistic sense. That gives us stuff like

$$P(\{X_{0} = 1\}\cap\{X_{1} = 1\} = P(X_{0} = 1)P(X_{1}=1)$$

and we can get the joint distribution by enumerating all the combinations of $X_{0}$ and $X_{1}$, as you've done. Nothing circular to be found.

After thinking about this some more, I believe the answer is entropy maximization. The entropy of the joint distribution is highest when the two coins are independent.

• The entropy can be regarded as the amount of missing information. Any known dependence or correlation between A and B reduces the amount of missing information. Thus, no correlation or dependence maximizes the entropy. Jun 29, 2023 at 13:21
• Yes. The important part is that entropy maximization is true in physics, explaining why the real world produces probabilities that match the arithmetic definition of independence.
– Owen
Jun 29, 2023 at 17:07