For 2 variables to be independent of each other, should the correlation = 0 or mutual information = 0 or covariance = 0. I have seen different conditions and all these are really confusing.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ What different conditions have you seen? $\endgroup$– Glen_bJul 30, 2014 at 11:52
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
covariance=0 implies correlation=0 (as long as the variances aren't 0).
correlation (or covariance 0) is necessary but not sufficient for independence. Independence implies both correlation and covariance are 0, but both can be 0 with perfectly dependent data.
See here:
If the variables are independent, Pearson's correlation coefficient is 0, but the converse is not true because the correlation coefficient detects only linear dependencies between two variables. For example, suppose the random variable X is symmetrically distributed about zero, and $Y = X^2$. Then Y is completely determined by X, so that X and Y are perfectly dependent, but their correlation is zero
As Dilip says, we need $E[|X^3|]$ to be finite for the specific example here; there will be similar criteria in other cases. For example, if $Y=f(X)$ for some even $f$, we'd need $E[|X.f(X)|]$ to be finite for the symmetry to make the covariance 0.
- As it says here:
I(X; Y) = 0 if and only if X and Y are independent random variables.
That is, mutual information 0 implies independence (and vice versa).
-
1$\begingroup$ Since you did open the can of worms of the variance possibly being $0$, you might want to close another can of worms by pointing out that the first highlighted statement needs the additional condition that $E[|X|^3]$ is finite. Else one cannot conclude that $E[XY]=E[X^3] = 0$ which is needed along the way towards asserting that cov$(X,Y)=0$. Better yet, since a single counterexample suffices to disprove a claimed result, let's just make $X$ a standard normal random variable or $U(-1,1)$ random variable and be done with it. $\endgroup$ Jun 29, 2015 at 11:22
-
1$\begingroup$ @Dilip The business with $E(|X^3|)\lt \infty$ is a red herring because it is introduced only to make this particular example rigorous. In fact, the finiteness condition is not needed at all because one could replace $X^2$ by any even, measurable nonlinear function of $X$ that has a finite expectation, such as $\min(X^2, a)$ for any constant $a$ with $\Pr(X\gt a)\lt 1/2$. $\endgroup$– whuber ♦Jun 29, 2015 at 14:54
-
1$\begingroup$ (+1) For completeness--since the Wikipedia reference doesn't prove the finial statement--note that Gibbs' Inequality (an easy consequence of Jensen's Inequality) shows that $I(X;Y)=0$ implies the KL-divergence between a marginal and the associated conditional distribution is always zero, immediately implying the independence of $X$ and $Y$. The key relationship is $$I(X;Y)=\mathbb{E}_Y(D_{KL}(p(x|y)\,||\,p(x))),$$ as shown at en.wikipedia.org/wiki/… ($p(x,y)$ is the joint distribution). $\endgroup$– whuber ♦Jun 29, 2015 at 15:08
-
$\begingroup$ @whuber I respectfully disagree that requiring $E[|X|^3]$ be finite is a red herring. As I said in my comment, all that is necessary to disprove the claim that perfectly dependent random variables cannot be uncorrelated (or have covariance $0$) is to produce just one example, and the Wikipedia entry fails at that job by just a tiny little bit. If you don't like the $E[|X|^3]<\infty$ part, why not just use $X\sim U(-1,1)$ which anyone can verify for themselves? and or take $X \sim N(0,\sigma^2)$ which requires a tad more work (or knowledge of moments of normal random variables)? $\endgroup$ Jun 29, 2015 at 21:47
-
$\begingroup$ @Dilip I think you actually are agreeing with me! Issues about expectations of third moments came up only in the process of producing a particular counterexample. Other counterexamples are fine, too, as you state--and they (similarly) don't need to deal with third moments. Emphasizing this third moment criterion thereby becomes an unnecessary distraction and might even confuse people into thinking it has something to do in general with correlation. $\endgroup$– whuber ♦Jun 29, 2015 at 21:51
$\begingroup$
$\endgroup$
2
Two variables are considered independent if they are orthogonal to each other. Which means that their dot product is equal to 0.
$\mathbf{a}\cdot \mathbf{b} = \sum_{i=1}^n a_ib_i = a_1b_1 + a_2b_2 + \cdots + a_nb_n = 0$
In other words one variable doesn't contain any information on the second variable.
Different conditions for independence come from different scientific fields like linear algebra or probability theory, but the underlying concept is always the same.
-
5$\begingroup$ "the underlying concept is always the same"? For random variables $X$ and $Y$, independence has nothing to do with orthogonality. $X\cdot Y$ may not even make sense. $\endgroup$ Jul 30, 2014 at 12:57
-
1$\begingroup$ It makes sense to define the dot product between two random variables as the expectation over the product. However, also in that case it is only related to the covariance and therefore only indirectly with independence. $\endgroup$– fabeeJul 30, 2014 at 17:24