2
$\begingroup$

Gaussian distributions appear everywhere in nature, indeed this was largely the justification for most classical methods' reliance on assumption of normality.

ICA assumes non-Gaussian sources, indeed it requires this assumption to hold for the derivation of components via Central Limit Theorem.

But what then is the value of a method that only works with assumptions contrary to the known reality? Note that unlike the many previous questions I am not asking why or how ICA requires non-Gaussianity. I understand and accept that ICA requires this. What I don't understand is how this doesn't destroy the perceived usefulness of ICA.

What if one wished to separate a mixture of Gaussian sources? What if one didn't know the distributions of the sources (as is typical), then it would seem the 'default assumption' should be for rather than against their Gaussianity - in this case a method that can only give non-Gaussian answers is surely misleading at best?

Are scientists really just 'flip-flopping' on how Gaussian they assume the world to be when they use ICA versus classical methods?

Improve this question
$\endgroup$
2
  • $\begingroup$ Not familiar with ICA enough to provide an answer so commenting instead: Isn't the point that the mixture of two Gaussians (with different means) is non-gaussian? So in this case the ICA would find a projection that separates the two Gaussians, because two Gaussians added together give non-Gaussian distribution. $\endgroup$
    – Karolis Koncevičius
    Jun 20, 2018 at 12:05
  • $\begingroup$ @KarolisKoncevičius I understood it to be the other way round: the sum of independent random variables is more Gaussian than the variables themselves as per CLT. ICA therefore seeks to reverse this. A quick Google gives: mfviz.com/central-limit ; matlabtips.com/ica-demystified . You highlight my point though, that ICA seemingly has little to say about the case where the random variables being summed are themselves Gaussian (and this is surely the default/typical case) $\endgroup$
    – benxyzzy
    Jun 20, 2018 at 12:12

1 Answer 1

Reset to default
6
$\begingroup$

ICA essentially reverses the central limit theorem: if averages (or generally speaking: linear combinations) of data becomes more and more Gaussuan, then one way to uncover the original signals from a linear combination of unknown sources is to look for those that are most non-Gaussian. The more non-Gaussian data you find, the closer to the source you should be.

ICA for all Gaussian signals won't work. Use PCA then, also since co-variance structure is all you need to know anyways about the data if its multivariate Gaussian.

Re Gaussianity: An interesting article i can recommend is "Why Gaussianity?" (https://ieeexplore.ieee.org/document/4472249/)

One takeaway is that Gaussianity is the max entropy distribution for given mean and variance. So the reason we often assume it in methodologies/models is because we want to assume a world under most uncertainty possible for any method to (still) work.

Re reality is Gaussian: While averages become more Gaussian, "reality" is in fact not (!) Gaussian (speech, income, # of people you know, stock market, color distribution in images, word frequency, music, traffic delay times, temperature, neural brain activity, seismic activity, ecg heart signals, ...). So most of the times you see Gaussian in 'reality' it's either averages / combinations of other signals (-> ICA comes in handy) or brownian motion - i doubt you see the latter very often ;).

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for the link, it sounds enlightening. I always assumed that the abundance of measured Gaussian distributions comes from the fact the quantity you're measuring is usually some composite of the effects of yet deeper, more hidden/fundamental processes. If so, then you are saying ICA would 'cut through' that level of measurable Gaussian variables to the non-Gaussian fundamentals? And you are saying the necessarily non-Gaussian nature of these as returned by ICA is not a misrepresentation of nature? $\endgroup$
    – benxyzzy
    Jun 20, 2018 at 12:34
  • $\begingroup$ In other words, take for example height whose mean varies between populations (gender, age, nutrition...) but is normally distributed within each. If I had an aggregate population with many genders, ages or nutritional histories, ICA on the heights of these could not deliver the component subpopulations but would instead attempt to separate the myriad biological and environmental factors that contribute to height? $\endgroup$
    – benxyzzy
    Jun 20, 2018 at 12:39
  • $\begingroup$ re "cut through": I'm by no means an ICA expert, but yes that's one way I would interpret what ICA does (e.g., when applied to the cocktail party problem of separating speech signals; or when applied to stock market data to find underlying 'shocks', or applied to brain activity data to find information exchange, ...). re height example: not sure I follow what the data looks like. ICA is applied to a multivariate signal; you can't apply ICA to one signal ("height") to uncover subpopulations (use a mixture distribution or clustering for that). $\endgroup$
    – Georg M. Goerg
    Jun 20, 2018 at 12:48
  • $\begingroup$ Yes it was a confused example, but I think it works if you consider each individual('s height) to be a variable... $\endgroup$
    – benxyzzy
    Jun 20, 2018 at 13:07
  • $\begingroup$ what are your measurements then? Over time? it's still not clear to me what the "X" matrix (N rows, p columns) represents: height of p individuals measured over time (N time points)? $\endgroup$
    – Georg M. Goerg
    Jun 20, 2018 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.