10
$\begingroup$

I know it's commonly asked why Gaussians are forbidden from use in independent components analysis. This is because a gaussian source distribution will result in the same observed distribution no matter what the mixing matrix $A$ is. Hence, it is impossible to determine $A$ and thus impossible to find the sources $s$ in any useful way. This isn't my question, though. Sources almost always point out that the multivariate Gaussian with mean zero and covariance $I$ is rotationally symmetric or rotationally invariant. This point correlates so strongly that someone (like me) who hasn't worked out the math explicitly might conclude that the two facts are equivalent.

I suppose I have two questions.

First, is the fact that zero-mean identity-covariance (or any matrix $aI$) Gaussians are rotationally symmetric sufficient to preclude the use of Gaussians from ICA?

Second, and more interestingly, if we consider distributions of the form:

$$p(s) = \prod_{i} p_s(s_i)$$

(i.e., independent components) is a Gaussian the only form $p_s$ can take that will render $p(s)$ rotationally symmetric? Why? If true, this seems like a crazy fact to me, yet it's not mentioned here, and it seems like many distributions (like a logistic distribution centered at 0) would have this effect.


Edit: I think I may be getting closer to answer this for myself. Part of the difficulty is the temptation of visualizing distributions of the form $p(s) = \prod_{i} p_s(s_i)$ as volumes of revolutions of $p_s$. But clearly this is not the case (e.g., $p_s = U[-1, 1]$). So I've decided to resist trying to "picture it" altogether.

Further, if rotational invariance is sufficient for a distribution to be "forbidden" from ICA, it seems that the critical property is that the probability of some $\mathbf{s}$ is that it depends only on its distance from the origin. This means that "forbidden" distributions $p_s$ are distributions that have the property (for some length $N$ vector $\mathbf{s}$):

$$p_s(\| \mathbf{s} \|_2)p_s(0)^{N-1} = \prod_{i=1}^N p_s(s_i)$$

where N is the number of elements of $\mathbf{s}$. This is true for the standard normal distribution, and not true for logistic distributions (the distribution I asked about above), for instance one where $\mu=0$ and $s=1$. If $N=2$:

$$\frac{1}{4}\frac{e^{-\sqrt{a^2+b^2}}}{(1+e^{-\sqrt{a^2+b^2}})^2} \neq \frac{e^{-(a+b)}}{(1+e^{-a})^2(1+e^{-b})^2}$$

Is it only Gaussians that have this property?

$\endgroup$

1 Answer 1

2
$\begingroup$

First question: No. There exist non-Gaussian rotationally-invariant distributions (for example, a multivariate student-t distribution).

Second question: Yes, the only rotationally symmetric distributions with independnt components are spherical Gaussians. This is sometimes called Maxwell's theorem, and I agree it is a crazy fact.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.