# How do we understand the relationship between independent probabilities and real-world independence?

From what I have come to understand, the events A and B are considered independent for purposes of probability theory when

$$p(A \cap B) = p(A) \cdot p(B)$$

Now, supposing I flip two coins. I write down the probabilities for the joint outcomes as $\left(\frac{1}{4}, \frac{1}{4}, \frac{1}{4}, \frac{1}{4}\right)$ and see that the individual coins are independent. But how do we know those probabilities? Well, the first coin could come up either heads or tails with probability $\frac{1}{2}$, then in the case of heads we know that the probabilities for the second coin are still $\frac{1}{2}$… but this seems to be assuming $p(A|B) = p(A)$, which is mathematically just a rearrangement of the above formula, so this feels like circular reasoning.

So, what's the explanation? Why does theoretical independence correspond to practical independence, in a not-apparently-circular way?

• Real world independence?. When occurrence and non occurrence of a particular object does not depend on occurrence and non occurrence of another object is called the independence of these objects. In real world where this definition stand statistical independence and real world independence are same. – SAAN Aug 30 '13 at 8:56
• By 'real world independence' do you mean something like 'physically independent' or something else? Independence in the probability sense is a jargon term with a very specific meaning which applies to probability models -- mathematical constructs which needn't relate to anything in the real world at all. – Glen_b -Reinstate Monica Aug 30 '13 at 9:59
• You've assumed independence when constructing the joint distribution. – jtobin Aug 30 '13 at 10:06
• In the real world the impact of the lack of complete independence (things like the butterfly effect) is probably less substantial than the lack of precise fairness, i.e. the probability of heads is almost certainly not exactly 1/2. – Henry Aug 30 '13 at 10:20
• I think you need to expand on what you mean by "circularity" in your penultimate paragraph. Let X and Y be Bernouilli random variables for coin flips and writing h for head and t for tail, we have P(X=h)=P(X=t)=½, and also P(Y=h)=P(Y=t)=½. By the first formula you quoted, P(X=x,Y=y) = P(X=x)P(Y=y), so P(X=h,Y=h) = P(X=h)P(Y=h) = ½ x ½ = 1/4. Similar reasoning yields 1/4 for the other probabilities. This doesn't assume P(A|B) = P(A), so there is no circularity. – TooTone Aug 30 '13 at 10:33

I think if you approach it the other way around it is more intuitively understandable. What does it mean that two variables $A$ and $B$ are independent? It means that knowing the probability distribution/function of one tells us absolutely nothing about the other one. The fact that $B$ may or may not have occurred is irrelevant, extraneous, and pretty much distracting when we are thinking of $A$. What does this mean in probabilistic terms? It means that the probability of $A$ occurring, given that $B$ occurred is the same as the probability of $A$ ignoring $B$ completely. We translate this English (or whatever language you wish) concept into symbolic mathematical form as $P(A|B) = P(A)$. This is the root expression, not the product rule, as this is the translation into mathematical symbols of the meaning of the word "independent".

Now that we have defined/translated "independence" to mean $P(A|B) = P(A)$, we can make the following observation: $$P(A|B) = P(A)\\ \frac{P(A \cap B)}{P(B)} = P(A)\\ P(A \cap B) = P(A)P(B)$$ The "product rule" is an outgrowth (via algebraic manipulation) of the definition, not the other way around.

• What you say makes sense intuitively and is a really nice way of explaning it, but I quibble on "not the other way round". In fact I have seen independence defined via $P(A\cap B)=P(A)P(B)$, and for pdfs, $f_{XY}(x,y)=f_X(x)f_Y(y)$. In the discrete case, starting from (1) $P(A\cap B) = P(A)P(B)$, and the definition of conditional probability (2) $P(A|B)=P(A\cap B)/P(B)$, you can then substitute (1) into (2) to get $P(A|B) = P(A)$. – TooTone Aug 30 '13 at 16:48
• I've seen it that way too, but I harken back to my bible on Probability, DeGroot 2ed, and where he introduces the concept of independence (ch. 1.11), the explanations are of the form (paraphrased p. 44) since X and Y are unrelated, the sample space is the product of the individual outcomes *each equally possible*… Which is just the way to say in English $P(A|B) = P(A)$, so I think that even if the independence is defined as the product rule, it is always justified (sometimes only implicitly) using the conditional definition. (Larson 1982 pp.70-71) defines it explicitly using conditionals. – Avraham Aug 30 '13 at 17:01
• I like your way of explaining it, but I do not think that this answers my question. Why is the translation in question (from colloquial "provides no information" to the equation) a faithful one? That is really what I am trying to figure out. – Owen Aug 30 '13 at 23:50
• Your explanation is nice but it runs into difficulties when $P(B)=0$. The definition and concept of conditional independence are complicated and difficult in this case, whereas a formula like $P(A\cap B) =P(A)P(B)$ is simple and clear. – whuber Aug 31 '13 at 2:48

I think there's a problem where you say you "observe the two coins are independent." You never observe probabilistic independence, per se; it is always a property of the events / random variables under consideration, which are constructed by the modeller to represent some (physical or otherwise) phenomenon.

So, the physical phenomenon here is: there are two fair coins, and tossing either one of them does not affect the outcome of the other. They're 'independent' in a nontechnical sense, if you want to use that word.

A probabilistic model that we can build to represent this physical phenomenon mathematically is: define two independent, Bernoulli(1/2)-distributed random variables $X_{0}$ and $X_{1}$. I.e.,

$$X_{0}, X_{1} \overset{iid}\sim \text{bernoulli}(1/2) \\$$

Here I mean 'independent' in the technical, probabilistic sense. That gives us stuff like

$$P(\{X_{0} = 1\}\cap\{X_{1} = 1\} = P(X_{0} = 1)P(X_{1}=1)$$

and we can get the joint distribution by enumerating all the combinations of $X_{0}$ and $X_{1}$, as you've done. Nothing circular to be found.