2
$\begingroup$

It is given in the following image that

lack of correlation determines the second-degree cross-moments (covariances) of a multivariate distribution,while in general statistical independence determines all of the cross moments

It is not making any sense to me. Please explain it.

It has been taken from the book Elements of Statistical Learning chapter Unsupervised Learning. enter image description here enter image description here

$\endgroup$

2 Answers 2

2
$\begingroup$

The components of a Multivariate Normal distribution are independent if and only if they are uncorrelated. So cross-moments of higher than 2nd order don't come into play for independence determination.

For a general multivariate distribution, independence of components implies uncoorrelatedness of the components, but uncoorrelatedness of the components does not imply independence of the components. So cross-moments of higher than 2nd order can come into play for independence determination.

$\endgroup$
2
  • $\begingroup$ Sir, first paragraph is not clear to me . Would you explain it a little bit more with examples. thanks $\endgroup$
    – ironman
    Jun 15, 2018 at 12:02
  • 1
    $\begingroup$ Consider the covariance matrix of a Multivariate Normal. If all the off-diagonal elements are zero, the components are independent.. For a general multivariate distribution, it is possible that all off-diagonal elements of its coovariace matrix are zero, but the components are not independent. See stats.stackexchange.com/questions/85363/… for examples of this. $\endgroup$ Jun 15, 2018 at 12:09
0
$\begingroup$

If $S \in \mathbb{R}^q$ is multivariate standard normal, then for $\mathbf{A} \in \mathbb{R}^{p \times q}$ we have

$$X = \mathbf{A}S$$

is multivariate normal.

Speaking intuitively, there is not much "freedom" in what the cross-moments are allowed to be if your distributions are multivariate normal. The distribution of $S$ are fixed by its covariance matrix $\mathbf{I}$, which consists of second-order (cross) moments, which are proportional to correlations. Once uncorrelatedness has specified these moments, we've already achieved independence; then all the higher order cross moments are automatically determined in terms of the standard deviations $\sigma = 1$, via (see Wikipedia):

$$\mathbb{E}[S_1^n S_2^m] = \mathbb{E}[S_1^n] \mathbb{E}[S_2^m] = \sigma \cdot (n - 1)!! \cdot \sigma \cdot (m - 1)!! $$

if $n$ and $m$ are odd.

If $S$ were independent but not multivariate normal, then $S$ would not rely only on its covariance matrix to determine its distribution. We can specify uncorrelatedness (second moments), but unlike in the Gaussian case where the second moments contain all the information, we can "keep going" and specify higher cross moments of $S$ to approach independence. Since the higher moments aren't fixed automatically by the second moments, there is some additional information contained in them.

So intuitively there should be some ambiguity in the choice of $\mathbf{A}$ in the Gaussian case, since specifying independence means specifying uncorrelatedness means we only get information from the second moments. This is not rigorous, but the ambiguity seems to arise via us freely being able to replace $\mathbf{A}$ with $\mathbf{A}\mathbf{R}^T$ for any orthogonal $\mathbf{R}$.

When $S$ is not Gaussian, the additional information contained in the higher moments is what intuitively fixes the choice for $\mathbf{A}$.

=============

some tidbits:

In your book you can consider equation (14.85). We would like to minimize the entropy with respect to $\mathbf{A}$, but if the distributions are Gaussian, because they have equal variance their entropy is fixed and we cannot adjust it. We need non-Gaussianity to give us more information to bring it lower.

The explanation given by Hyvarinen and Oja (2000), which the book cites as a resource on page 560, is simpler. It instead says that the standard multivariate Gaussian doesn't work because it is rotationally symmetric. Indeed, replacing $S$ with $\mathbf{R}S$ leaves the entire distribution of $X$ unchanged, for any rotationally symmetric distribution for $S$. But it still turns out $S$ must be Gaussian, due to a theorem which says that if $S$ is rotationally symmetric with independent components then it must be Gaussian.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.