1
$\begingroup$

Suppose I have a dataset that was sampled from a mixture of Gaussians: $$ X \sim \sum_i w_i \mathcal{N}(\mu_i, \Sigma_i)$$

Technically, I can center and then whiten $X$ so that it has zero mean and unit covariance. But something about that process seems unsatisfactorily oversimplified: for example, what if the means of the mixture were arranged along a semicircle, such that no values in $X$ actually occurred at the overall sample mean ?

It seems like this is a case where whitening wouldn't really have much meaning in a global sense. However, given only a dataset, one could theoretically fit a mixture of Gaussians model to it, and then use the estimated covariances to whiten points in local regions of the data space.

My question is, has this process already been formalized somehow under some name that I have yet to come across? Does it even make sense? What are some good references I ought to consult to learn more about "local whitening"?

$\endgroup$
  • 2
    $\begingroup$ What do you mean exactly by "whitening"? Does it have anything to do with white noise? Or with transformation of variables, so they appear uncorrelated? $\endgroup$ – Adam Ryczkowski Sep 2 '13 at 22:02
  • $\begingroup$ @AdamRyczkowski I was thinking of it more in the second sense (though the two are related) -- that's why I described the whitened $X$ as having unit covariance. Should I clarify that somehow ? $\endgroup$ – lmjohns3 Sep 3 '13 at 3:33
  • 2
    $\begingroup$ Well, I still some more explanation. $\endgroup$ – Adam Ryczkowski Sep 3 '13 at 5:51
  • $\begingroup$ Whitening is a little unclear here. There is no sense of time in the problem statement, so I wouldn't know how to cast this in terms of a white noise signal. It seems like we are just talking about 'standardising' a sample of data. Which we can do, but what sort of meaning are we looking for in doing that? $\endgroup$ – conjectures Mar 7 '14 at 9:57
  • $\begingroup$ @conjectures Yes, I meant whitening in the sense of decorrelating a dataset. As for "what sort of meaning" -- that's basically what I'm trying to ask! In machine learning, whitening is often used as a preprocessing step, even when our model explicitly assumes an underlying distribution that is non-Gaussian (e.g., RICA, "manifold learning" using autoencoders, etc.). I'm curious whether anyone's looked at "local whitening" already. $\endgroup$ – lmjohns3 Feb 1 '15 at 18:21
2
$\begingroup$

If you go all probabilistic, your approach to whitening would be the assumption that your data is generated from a standard normal Gaussian through some map $f$. Now, if $f$ is linear, you can use techniques such as ZCA or PCA to find the latent representations. As you already know, such $f$ cannot turn a standard normal into a mixture--you need more expressive models for that, which comes down to find some different class for $f$ which you can also estimate.

One such class would be a deep latent gaussian model.

$\endgroup$
1
$\begingroup$

Keep in mind, that whitening will preserve the semicircle shape of the data, so it seems irrelevant to your problem of fitting mixture of Gaussians.

On the other hand, as you should already know, fitting mixtures of any distributions is difficult task, because there is never a guarantee, that fitted solution is globally-optimal. You can achieve very similar shape of the overall mixture with very different mixtures of Gaussians.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.