PCA of non-Gaussian data

Question

I have a couple of quick questions about PCA:

• Does the PCA assume that the dataset is Gaussian?
• What happens when I apply a PCA to inherently non-linear data?

Given a dataset, the process is to first mean-normalize, set the variance to 1, take an SVD, reduce rank, and finally map the dataset into the new reduced-rank space. In the new space, each dimension corresponds to a "direction" of maximal variance.

• But is the correlation of that dataset in the new space always zero, or is that only true for data that is inherently Gaussian?

Suppose I have two datasets, "A" and "B", where "A" corresponds to randomly sampled points taken from a Gaussian, while "B" corresponds to points randomly sampled from another distribution (say Poisson).

• How does the PCA(A) compare to the PCA(B)?
• By looking at the points in the new space, how would I determine that the PCA(A) corresponds to points sampled from a Gaussian, while PCA(B) corresponds to points sampled from a Poisson?
• Is the correlation of the points in "A" 0?
• Is the correlation of points in "B" also 0?
• More importantly, am I asking the "right" question?
• Should I look at the correlation, or is there another metric that I should consider?

Answer 1

You have a couple of good answers here already (+1 to both @Cam.Davidson.Pilon & @MichaelChernick). Let me throw out a couple of points that help me to think about this issue.

First, PCA operates over the correlation matrix. Thus, it seems to me the important question is whether it makes sense to use a correlation matrix to help you think about your data. For example, the Pearson product-moment correlation assesses the linear relationship between two variables; if your variables are related, but not linearly, the correlation is not an ideal metric to index the strength of the relationship. (Here is a nice discussion on CV about correlation and non-normal data.)

Second, I think the easiest way to understand what is going on with PCA is that you are simply rotating your axes. You can do more things, of course, and unfortunately PCA gets confused with factor analysis (which definitely does have more going on). Nevertheless, plain old PCA with no bells and whistles, can be thought of as follows:

• you have some points plotted in two dimensions on a sheet of graph paper;
• you have a transparency with orthogonal axes drawn on it, and a pinhole at the origin;
• you center the origin of the transparency (i.e., the pinhole) over $(\bar x, \bar y)$ and put the tip of your pencil through the pinhole to hold it in place;
• then you rotate the transparency until the points (when indexed according to the transparency's axes instead of the original ones) are uncorrelated.

This isn't a perfect metaphor for PCA (e.g., we didn't rescale the variances to 1). But does give people the basic idea. The point is now to use that image to think about what the result looks like if the data weren't Gaussian to begin with; that will help you decide whether this process was worth doing. Hope that helps.

Answer 2

I can give a partial solution and show an answer for your second paragraph third question, relating to whether the new data is correlated. The short answer is no, the data in the new space is not correlated. To see, consider $w_1$ and $w_2$ as two unique principle components. Then $Xw_1$ and $Xw_2$ are two dimensions in the new space of the data, $X$.

$$ {\rm Cov}( Xw_1, Xw_2 ) = E[ (Xw_1)^T(Xw_2) ] - E[Xw_1]^TE[Xw_2] $$ As $w_i$ are constant, the second term is 0 (as you said we demean $X$ prior). The first term can be rewritten as $$ w_1^TE[X^TX]w_2 = {\rm Var}(X)w_1^Tw_2 = 0$$ as $w_i$ are orthonormal to each other, so the whole term is zero, assuming $Var(X)$ is finite. This was all independent of any assumption about normality.

I think the reliance on normality boils down to the whole debate over variance. Here's a intuitive argument: First, note that variance is a really good measure of "spread" for symmetric distributions. But it can fail when we consider skewed or asymmetric distributions. Now recall that PCA tries to maximize the variance in the projected dimension. If $X$ is normal, then $Xw$ is still normal, i.e. still symmetric and variance works well. But if $X$ is not normal, like Poisson, the variance of $Xw$ need not be very descriptive.

To give an example where variance (and standard deviation) break down, consider the pareto distribution. The variance drops quickly as $\alpha$ grows, but only because the data starts to group around the small mean. But we know that we can easily see large swings with the pareto distribution, something that a small variance would not describe well.

Answer 3

There is no linearity or normality assumed in PCA. The idea is just decomposing the variation in a p-dimensional dataset into orthogonal components that are ordered according to amount of variance explained.

Answer 4

Reading page 7 here:

http://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf

they note that PCA assumes that the distribution of whatever we are explaining can be described by a mean (of zero) and variance alone, which they say can only be the Normal distribution.

(Basically in addition to Cam's answer, but I don't have enough reputation to comment : )

Answer 5

As far as I know, PCA doesn't assume normality of data. But if it is normally distributed (in more general sense, symmetrically distributed), then the result is more robust. As other people say, the key is that PCA is based on Pearson correlation coefficient matrix, of which estimation is affected by outliers and skewed distribution. So in some analysis involved in, such as statistical test or p-value, then you should care more about whether normality is satisfied; but in other applications like exploratory analysis, you can use it but only take care when make interpretations.

Answer 6

Agreed with others who said data should be "Normally" distributed. Any distribution will overlap with a normal distribution if you transform it. If your distribution is not normal, the results you will get will be inferior compared to the case when it is normal, as stated by some here...

• You can transform your distribution if you need.
• You can opt the PCA and use Independent Component Analysis (ICA) instead.

If you read the reference in the first answer, in the Appendix section it states that the assumption is a Normal distribution.