20
$\begingroup$

I have a couple of quick questions about PCA:

  • Does the PCA assume that the dataset is Gaussian?
  • What happens when I apply a PCA to inherently non-linear data?

Given a dataset, the process is to first mean-normalize, set the variance to 1, take an SVD, reduce rank, and finally map the dataset into the new reduced-rank space. In the new space, each dimension corresponds to a "direction" of maximal variance.

  • But is the correlation of that dataset in the new space always zero, or is that only true for data that is inherently Gaussian?

Suppose I have two datasets, "A" and "B", where "A" corresponds to randomly sampled points taken from a Gaussian, while "B" corresponds to points randomly sampled from another distribution (say Poisson).

  • How does the PCA(A) compare to the PCA(B)?
  • By looking at the points in the new space, how would I determine that the PCA(A) corresponds to points sampled from a Gaussian, while PCA(B) corresponds to points sampled from a Poisson?
  • Is the correlation of the points in "A" 0?
  • Is the correlation of points in "B" also 0?
  • More importantly, am I asking the "right" question?
  • Should I look at the correlation, or is there another metric that I should consider?
$\endgroup$
  • 2
    $\begingroup$ See the appendix on assumptions of PCA in this paper. $\endgroup$ – assumednormal Jul 11 '12 at 14:27
18
$\begingroup$

You have a couple of good answers here already (+1 to both @Cam.Davidson.Pilon & @MichaelChernick). Let me throw out a couple of points that help me to think about this issue.

First, PCA operates over the correlation matrix. Thus, it seems to me the important question is whether it makes sense to use a correlation matrix to help you think about your data. For example, the Pearson product-moment correlation assesses the linear relationship between two variables; if your variables are related, but not linearly, the correlation is not an ideal metric to index the strength of the relationship. (Here is a nice discussion on CV about correlation and non-normal data.)

Second, I think the easiest way to understand what is going on with PCA is that you are simply rotating your axes. You can do more things, of course, and unfortunately PCA gets confused with factor analysis (which definitely does have more going on). Nevertheless, plain old PCA with no bells and whistles, can be thought of as follows:

  • you have some points plotted in two dimensions on a sheet of graph paper;
  • you have a transparency with orthogonal axes drawn on it, and a pinhole at the origin;
  • you center the origin of the transparency (i.e., the pinhole) over $(\bar x, \bar y)$ and put the tip of your pencil through the pinhole to hold it in place;
  • then you rotate the transparency until the points (when indexed according to the transparency's axes instead of the original ones) are uncorrelated.

This isn't a perfect metaphor for PCA (e.g., we didn't rescale the variances to 1). But does give people the basic idea. The point is now to use that image to think about what the result looks like if the data weren't Gaussian to begin with; that will help you decide whether this process was worth doing. Hope that helps.

$\endgroup$
  • 2
    $\begingroup$ +1 (long time ago). I think this is the best answer in this thread, hope it will gather one more upvote to become the most upvoted one too. I like your way of explaining PCA with a transparency, that's nice. $\endgroup$ – amoeba Jan 29 '15 at 10:55
  • $\begingroup$ By the way, this answer of yours inspired my recent answer in our huge layman PCA thread: I made those animated gifs having your transparency analogy in mind. $\endgroup$ – amoeba Mar 14 '15 at 0:14
  • $\begingroup$ That's a great answer, @amoeba. It's much better than this is. $\endgroup$ – gung Mar 14 '15 at 3:52
12
$\begingroup$

I can give a partial solution and show an answer for your second paragraph third question, relating to whether the new data is correlated. The short answer is no, the data in the new space is not correlated. To see, consider $w_1$ and $w_2$ as two unique principle components. Then $Xw_1$ and $Xw_2$ are two dimensions in the new space of the data, $X$.

$$ {\rm Cov}( Xw_1, Xw_2 ) = E[ (Xw_1)^T(Xw_2) ] - E[Xw_1]^TE[Xw_2] $$ As $w_i$ are constant, the second term is 0 (as you said we demean $X$ prior). The first term can be rewritten as $$ w_1^TE[X^TX]w_2 = {\rm Var}(X)w_1^Tw_2 = 0$$ as $w_i$ are orthonormal to each other, so the whole term is zero, assuming $Var(X)$ is finite. This was all independent of any assumption about normality.

I think the reliance on normality boils down to the whole debate over variance. Here's a intuitive argument: First, note that variance is a really good measure of "spread" for symmetric distributions. But it can fail when we consider skewed or asymmetric distributions. Now recall that PCA tries to maximize the variance in the projected dimension. If $X$ is normal, then $Xw$ is still normal, i.e. still symmetric and variance works well. But if $X$ is not normal, like Poisson, the variance of $Xw$ need not be very descriptive.

To give an example where variance (and standard deviation) break down, consider the pareto distribution. The variance drops quickly as $\alpha$ grows, but only because the data starts to group around the small mean. But we know that we can easily see large swings with the pareto distribution, something that a small variance would not describe well.

$\endgroup$
7
$\begingroup$

There is no linearity or normality assumed in PCA. The idea is just decomposing the variation in a p-dimensional dataset into orthogonal components that are ordered according to amount of variance explained.

$\endgroup$
  • 2
    $\begingroup$ True but "decomposing the variation in a p-dimensional dataset into orthogonal components" isn't very useful when there are non-linear dependencies between the variables since the orthogonalization was usually done so that you can argue that the dimensions are unrelated (which is also related to the Gaussian part of the question). When you're doing PCA and plan to interpret the results in the usual way, there's an underlying assumption that the data lives in a lower dimensional linear subspace. $\endgroup$ – Macro Jul 11 '12 at 15:41
  • 2
    $\begingroup$ @Macro Not exactly. I would say that the underlying assumption is that at least most of the variability and hence pattern of the data is concentrated in some lower dimensional space. I can view a parabola very well in a 2-dimensional space with orthogonal components. I think nonlinear shapes can be viewed in two or three dimensions. If the data come from a multivariate Gaussian distrbution then in some subspace the points should look like an ellipsoidal cloud. The distribution does not have to look like an ellipsoid for its view in the subspace of the high PCs to be interesting. $\endgroup$ – Michael Chernick Jul 11 '12 at 15:48
  • 4
    $\begingroup$ I would qualify this slightly. There is no normality assumption in classical PCA or PCA by SVD. However, EM algorithms to compute PCA with missing data will assume normality and linearity. $\endgroup$ – John Jul 11 '12 at 16:39
  • $\begingroup$ While the classical road to PCA does not need any assumptions, there is another road to its solution which does: probabilistic PCA with 0 measurement noise. $\endgroup$ – bayerj Jan 29 '15 at 10:26
3
$\begingroup$

Reading page 7 here:

http://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf

they note that PCA assumes that the distribution of whatever we are explaining can be described by a mean (of zero) and variance alone, which they say can only be the Normal distribution.

(Basically in addition to Cam's answer, but I don't have enough reputation to comment : )

$\endgroup$
  • 1
    $\begingroup$ The link you provided to Shlens' tutorial is to version 1 of the tutorial, but version 3.02 (the final version?) is now available, and this specific point was removed. Also, this question asked exactly about that. $\endgroup$ – Oren Milman Aug 18 '18 at 15:18
0
$\begingroup$

As far as I know, PCA doesn't assume normality of data. But if it is normally distributed (in more general sense, symmetrically distributed), then the result is more robust. As other people say, the key is that PCA is based on Pearson correlation coefficient matrix, of which estimation is affected by outliers and skewed distribution. So in some analysis involved in, such as statistical test or p-value, then you should care more about whether normality is satisfied; but in other applications like exploratory analysis, you can use it but only take care when make interpretations.

$\endgroup$
-1
$\begingroup$

Agreed with others who said data should be "Normally" distributed. Any distribution will overlap with a normal distribution if you transform it. If your distribution is not normal, the results you will get will be inferior compared to the case when it is normal, as stated by some here...

  • You can transform your distribution if you need.
  • You can opt the PCA and use Independent Component Analysis (ICA) instead.

If you read the reference in the first answer, in the Appendix section it states that the assumption is a Normal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.