4
$\begingroup$

From what I have come to understand, the events A and B are considered independent for purposes of probability theory when

$$ p(A \cap B) = p(A) \cdot p(B) $$

Now, supposing I flip two coins. I write down the probabilities for the joint outcomes as $\left(\frac{1}{4}, \frac{1}{4}, \frac{1}{4}, \frac{1}{4}\right)$ and see that the individual coins are independent. But how do we know those probabilities? Well, the first coin could come up either heads or tails with probability $\frac{1}{2}$, then in the case of heads we know that the probabilities for the second coin are still $\frac{1}{2}$… but this seems to be assuming $p(A|B) = p(A)$, which is mathematically just a rearrangement of the above formula, so this feels like circular reasoning.

So, what's the explanation? Why does theoretical independence correspond to practical independence, in a not-apparently-circular way?

Improve this question
$\endgroup$
9
  • $\begingroup$ Real world independence?. When occurrence and non occurrence of a particular object does not depend on occurrence and non occurrence of another object is called the independence of these objects. In real world where this definition stand statistical independence and real world independence are same. $\endgroup$
    – SAAN
    Commented Aug 30, 2013 at 8:56
  • 3
    $\begingroup$ By 'real world independence' do you mean something like 'physically independent' or something else? Independence in the probability sense is a jargon term with a very specific meaning which applies to probability models -- mathematical constructs which needn't relate to anything in the real world at all. $\endgroup$
    – Glen_b
    Commented Aug 30, 2013 at 9:59
  • $\begingroup$ You've assumed independence when constructing the joint distribution. $\endgroup$
    – jtobin
    Commented Aug 30, 2013 at 10:06
  • $\begingroup$ In the real world the impact of the lack of complete independence (things like the butterfly effect) is probably less substantial than the lack of precise fairness, i.e. the probability of heads is almost certainly not exactly 1/2. $\endgroup$
    – Henry
    Commented Aug 30, 2013 at 10:20
  • 1
    $\begingroup$ Would it be too far-fetched to suppose that people develop theories and concepts to describe what they observe, and therefore what they observe actually does tend sometimes to conform to those theories? :-) $\endgroup$
    – whuber ♦
    Commented Aug 31, 2013 at 2:50

3 Answers 3

Reset to default
4
$\begingroup$

I think if you approach it the other way around it is more intuitively understandable. What does it mean that two variables $A$ and $B$ are independent? It means that knowing the probability distribution/function of one tells us absolutely nothing about the other one. The fact that $B$ may or may not have occurred is irrelevant, extraneous, and pretty much distracting when we are thinking of $A$. What does this mean in probabilistic terms? It means that the probability of $A$ occurring, given that $B$ occurred is the same as the probability of $A$ ignoring $B$ completely. We translate this English (or whatever language you wish) concept into symbolic mathematical form as $P(A|B) = P(A)$. This is the root expression, not the product rule, as this is the translation into mathematical symbols of the meaning of the word "independent".

Now that we have defined/translated "independence" to mean $P(A|B) = P(A)$, we can make the following observation: $$ P(A|B) = P(A)\\ \frac{P(A \cap B)}{P(B)} = P(A)\\ P(A \cap B) = P(A)P(B) $$ The "product rule" is an outgrowth (via algebraic manipulation) of the definition, not the other way around.

Improve this answer
$\endgroup$
8
  • $\begingroup$ What you say makes sense intuitively and is a really nice way of explaning it, but I quibble on "not the other way round". In fact I have seen independence defined via $P(A\cap B)=P(A)P(B)$, and for pdfs, $f_{XY}(x,y)=f_X(x)f_Y(y)$. In the discrete case, starting from (1) $P(A\cap B) = P(A)P(B)$, and the definition of conditional probability (2) $P(A|B)=P(A\cap B)/P(B)$, you can then substitute (1) into (2) to get $P(A|B) = P(A)$. $\endgroup$
    – TooTone
    Commented Aug 30, 2013 at 16:48
  • $\begingroup$ I've seen it that way too, but I harken back to my bible on Probability, DeGroot 2ed, and where he introduces the concept of independence (ch. 1.11), the explanations are of the form (paraphrased p. 44) since X and Y are unrelated, the sample space is the product of the individual outcomes *each equally possible*… Which is just the way to say in English $P(A|B) = P(A)$, so I think that even if the independence is defined as the product rule, it is always justified (sometimes only implicitly) using the conditional definition. (Larson 1982 pp.70-71) defines it explicitly using conditionals. $\endgroup$
    – Avraham
    Commented Aug 30, 2013 at 17:01
  • $\begingroup$ @TooTone References for above comment: DeGroot, M. H. Probability and Statistics Addison-Wesley Publishing Company, 1986 and Larson, H. Introduction to Probability Theory and Statistical Inference Wiley, 1982 $\endgroup$
    – Avraham
    Commented Aug 30, 2013 at 17:16
  • $\begingroup$ I like your way of explaining it, but I do not think that this answers my question. Why is the translation in question (from colloquial "provides no information" to the equation) a faithful one? That is really what I am trying to figure out. $\endgroup$
    – Owen
    Commented Aug 30, 2013 at 23:50
  • 1
    $\begingroup$ Your explanation is nice but it runs into difficulties when $P(B)=0$. The definition and concept of conditional independence are complicated and difficult in this case, whereas a formula like $P(A\cap B) =P(A)P(B)$ is simple and clear. $\endgroup$
    – whuber ♦
    Commented Aug 31, 2013 at 2:48
3
$\begingroup$

I think there's a problem where you say you "observe the two coins are independent." You never observe probabilistic independence, per se; it is always a property of the events / random variables under consideration, which are constructed by the modeller to represent some (physical or otherwise) phenomenon.

So, the physical phenomenon here is: there are two fair coins, and tossing either one of them does not affect the outcome of the other. They're 'independent' in a nontechnical sense, if you want to use that word.

A probabilistic model that we can build to represent this physical phenomenon mathematically is: define two independent, Bernoulli(1/2)-distributed random variables $X_{0}$ and $X_{1}$. I.e.,

$$ X_{0}, X_{1} \overset{iid}\sim \text{bernoulli}(1/2) \\ $$

Here I mean 'independent' in the technical, probabilistic sense. That gives us stuff like

$$ P(\{X_{0} = 1\}\cap\{X_{1} = 1\} = P(X_{0} = 1)P(X_{1}=1) $$

and we can get the joint distribution by enumerating all the combinations of $X_{0}$ and $X_{1}$, as you've done. Nothing circular to be found.

Improve this answer
$\endgroup$
0
$\begingroup$

After thinking about this some more, I believe the answer is entropy maximization. The entropy of the joint distribution is highest when the two coins are independent.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The entropy can be regarded as the amount of missing information. Any known dependence or correlation between A and B reduces the amount of missing information. Thus, no correlation or dependence maximizes the entropy. $\endgroup$
    – Romke Bontekoe
    Commented Jun 29, 2023 at 13:21
  • $\begingroup$ Yes. The important part is that entropy maximization is true in physics, explaining why the real world produces probabilities that match the arithmetic definition of independence. $\endgroup$
    – Owen
    Commented Jun 29, 2023 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.