# PCA of non-Gaussian data

I have a couple of quick questions about PCA:

• Does the PCA assume that the dataset is Gaussian?
• What happens when I apply a PCA to inherently non-linear data?

Given a dataset, the process is to first mean-normalize, set the variance to 1, take an SVD, reduce rank, and finally map the dataset into the new reduced-rank space. In the new space, each dimension corresponds to a "direction" of maximal variance.

• But is the correlation of that dataset in the new space always zero, or is that only true for data that is inherently Gaussian?

Suppose I have two datasets, "A" and "B", where "A" corresponds to randomly sampled points taken from a Gaussian, while "B" corresponds to points randomly sampled from another distribution (say Poisson).

• How does the PCA(A) compare to the PCA(B)?
• By looking at the points in the new space, how would I determine that the PCA(A) corresponds to points sampled from a Gaussian, while PCA(B) corresponds to points sampled from a Poisson?
• Is the correlation of the points in "A" 0?
• Is the correlation of points in "B" also 0?
• More importantly, am I asking the "right" question?
• Should I look at the correlation, or is there another metric that I should consider?
-
See the appendix on assumptions of PCA in this paper. –  Max Jul 11 '12 at 14:27

There is no linearity or normality assumed in PCA. The idea is just decomposing the variation in a p-dimensional dataset into orthogonal components that are ordered according to amount of variance explained.

-
True but "decomposing the variation in a p-dimensional dataset into orthogonal components" isn't very useful when there are non-linear dependencies between the variables since the orthogonalization was usually done so that you can argue that the dimensions are unrelated (which is also related to the Gaussian part of the question). When you're doing PCA and plan to interpret the results in the usual way, there's an underlying assumption that the data lives in a lower dimensional linear subspace. –  Macro Jul 11 '12 at 15:41
@Macro Not exactly. I would say that the underlying assumption is that at least most of the variability and hence pattern of the data is concentrated in some lower dimensional space. I can view a parabola very well in a 2-dimensional space with orthogonal components. I think nonlinear shapes can be viewed in two or three dimensions. If the data come from a multivariate Gaussian distrbution then in some subspace the points should look like an ellipsoidal cloud. The distribution does not have to look like an ellipsoid for its view in the subspace of the high PCs to be interesting. –  Michael Chernick Jul 11 '12 at 15:48
I would qualify this slightly. There is no normality assumption in classical PCA or PCA by SVD. However, EM algorithms to compute PCA with missing data will assume normality and linearity. –  John Jul 11 '12 at 16:39
Okay @John Thanks –  Michael Chernick Jul 11 '12 at 16:45

I can give a partial solution and show an answer for your second paragraph third question, relating to whether the new data is correlated. The short answer is no, the data in the new space is not correlated. To see, consider $w_1$ and $w_2$ as two unique principle components. Then $Xw_1$ and $Xw_2$ are two dimensions in the new space of the data, $X$.

$$cov( Xw_1, Xw_2 ) = E[ (Xw_1)^T(Xw_2) ] - E[Xw_1]^TE[Xw_2]$$ As $w_i$ are constant, the second term is 0 (as you said we demean $X$ prior). The first term can be rewritten as $$w_1^TE[X^TX]w_2 = Var(X)w_1^Tw_2 = 0$$ as $w_i$ are orthonormal to each other, so the whole term is zero, assuming $Var(X)$ is finite. This was all independent of any assumption about normality.

I think the reliance on normality boils down to the whole debate over variance. Here's a intuitive argument: First, note that variance is a really good measure of "spread" for symmetric distributions. But it can fail when we consider skewed or asymmetric distributions. Now recall that PCA tries to maximize the variance in the projected dimension. If $X$ is normal, then $Xw$ is still normal, i.e. still symmetric and variance works well. But if $X$ is not normal, like Poisson, the variance of $Xw$ need not be very descriptive.

To give an example where variance (and standard deviation) break down, consider the pareto distribution. The variance drops quickly as $\alpha$ grows, but only because the data starts to group around the small mean. But we know that we can easily see large swings with the pareto distribution, something that a small variance would not describe well.

-

You have a couple of good answers here already (+1 to both @Cam.Davidson.Pilon & @MichaelChernick). Let me throw out a couple of points that help me to think about this issue.

First, PCA operates over the correlation matrix. Thus, it seems to me the important question is whether it makes sense to use a correlation matrix to help you think about your data. For example, the Pearson product-moment correlation assesses the linear relationship between two variables; if your variables are related, but not linearly, the correlation is not an ideal metric to index the strength of the relationship. (Here is a nice discussion on CV about correlation and non-normal data.)

Second, I think the easiest way to understand what is going on with PCA is that you are simply rotating your axes. You can do more things, of course, and unfortunately PCA gets confused with factor analysis (which definitely does have more going on). Nevertheless, plain old PCA with no bells and whistles, can be thought of as follows:

• you have some points plotted in two dimensions on a sheet of graph paper;
• you have a transparency with orthogonal axes drawn on it, and a pinhole at the origin;
• you center the origin of the transparency (i.e., the pinhole) over $(\bar x, \bar y)$ and put the tip of your pencil through the pinhole to hold it in place;
• then you rotate the transparency until the points (when indexed according to the transparency's axes instead of the original ones) are uncorrelated.

This isn't a perfect metaphor for PCA (e.g., we didn't rescale the variances to 1). But does give people the basic idea. The point is now to use that image to think about what the result looks like if the data weren't Gaussian to begin with; that will help you decide whether this process was worth doing. Hope that helps.

-