8
$\begingroup$

I know it's commonly asked why Gaussians are forbidden from use in independent components analysis. This is because a gaussian source distribution will result in the same observed distribution no matter what the mixing matrix $A$ is. Hence, it is impossible to determine $A$ and thus impossible to find the sources $s$ in any useful way. This isn't my question, though. Sources almost always point out that the multivariate Gaussian with mean zero and covariance $I$ is rotationally symmetric or rotationally invariant. This point correlates so strongly that someone (like me) who hasn't worked out the math explicitly might conclude that the two facts are equivalent.

I suppose I have two questions.

First, is the fact that zero-mean identity-covariance (or any matrix $aI$) Gaussians are rotationally symmetric sufficient to preclude the use of Gaussians from ICA?

Second, and more interestingly, if we consider distributions of the form:

$$p(s) = \prod_{i} p_s(s_i)$$

(i.e., independent components) is a Gaussian the only form $p_s$ can take that will render $p(s)$ rotationally symmetric? Why? If true, this seems like a crazy fact to me, yet it's not mentioned here, and it seems like many distributions (like a logistic distribution centered at 0) would have this effect.


Edit: I think I may be getting closer to answer this for myself. Part of the difficulty is the temptation of visualizing distributions of the form $p(s) = \prod_{i} p_s(s_i)$ as volumes of revolutions of $p_s$. But clearly this is not the case (e.g., $p_s = U[-1, 1]$). So I've decided to resist trying to "picture it" altogether.

Further, if rotational invariance is sufficient for a distribution to be "forbidden" from ICA, it seems that the critical property is that the probability of some $\mathbf{s}$ is that it depends only on its distance from the origin. This means that "forbidden" distributions $p_s$ are distributions that have the property (for some length $N$ vector $\mathbf{s}$):

$$p_s(\| \mathbf{s} \|_2)p_s(0)^{N-1} = \prod_{i=1}^N p_s(s_i)$$

where N is the number of elements of $\mathbf{s}$. This is true for the standard normal distribution, and not true for logistic distributions (the distribution I asked about above), for instance one where $\mu=0$ and $s=1$. If $N=2$:

$$\frac{1}{4}\frac{e^{-\sqrt{a^2+b^2}}}{(1+e^{-\sqrt{a^2+b^2}})^2} \neq \frac{e^{-(a+b)}}{(1+e^{-a})^2(1+e^{-b})^2}$$

Is it only Gaussians that have this property?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.