25
$\begingroup$

I know that in a regression situation, if you have a set of highly correlated variables this is usually "bad" because of the instability in the estimated coefficients (variance goes toward infinity as determinant goes towards zero).

My question is whether this "badness" persists in a PCA situation. Do the coefficients/loadings/weights/eigenvectors for any particular PC become unstable/arbitrary/non-unique as the covariance matrix becomes singular? I am particularly interested in the case where only the first principal component is retained, and all others are dismissed as "noise" or "something else" or "unimportant".

I don't think that it does, because you will just be left with a few principal components which have zero, or close to zero variance.

Easy to see this isn't the case in the simple extreme case with 2 variables - suppose they are perfectly correlated. Then the first PC will be the exact linear relationship, and the second PC will be perpindicular to the first PC, with all PC values equal to zero for all observations (i.e. zero variance). Wondering if its more general.

$\endgroup$
  • 8
    $\begingroup$ Your reasoning is good. Actually, one would expect instability to occur when two or more eigenvalues are nearly coincident, for then although the eigenvalues are determined, the eigenvectors are not, and therefore neither are the loadings. For numerical reasons, there is also instability in eigenvalues (and eigenvectors) that are very small in size compared to the maximum eigenvalue. $\endgroup$ – whuber Apr 14 '11 at 5:34
  • $\begingroup$ @whuber comment answers your question, but I would like to note that in case of 2 perfectly correlated variables, the PCA should not have any problems. The covariance matrix would be of rank 1, so there will be only 1 non-zero eigenvalue, hence only 1 PC. The original variables will be the multiples of the this PC. The only issue may be the numerical stability. $\endgroup$ – mpiktas Apr 14 '11 at 11:13
  • $\begingroup$ In fact, I think you'd be worse off if you had moderately correlated variables than when you've got really highly correlated variables. Numerical-wise too, if you're using an algorithm like NIPALS that removes PC's in order $\endgroup$ – JMS Apr 16 '11 at 2:00
  • $\begingroup$ One thing - "highly correlated" and "colinear" are not the same. If there are more than 2 variables involved, colinearity does not imply correlation. $\endgroup$ – Peter Flom Aug 27 '11 at 18:33
11
$\begingroup$

The answer might be given in even simpler terms: the multiple regression has one step more than the pca if seen in terms of linear algebra, and from the second step the instability comes into existence:

The first step of pca and mult. regression can be seen as factoring of the correlation-matrix $R$ into two cholesky factors $L \cdot L^t$ , which are triangular -and which is indifferent to low or high correlations. (The pca can then be seen as a rotation of that (triangular) cholesky-factor to pc-position (this is called Jacobi-rotation as far as I remember)

The mult. regression procedure is the to apply an inversion of that cholesky factor $L$ minus the row and column of the dependent variable, which is conveniently in the last row of the correlation-matrix.
The instability comes into play here: if the independent variables are highly correlated, then the diagonal of the cholesky factor $L$ can degenerate to very small numeric values - and to invert that introduces then the problem of division by nearly-zero.

$\endgroup$
  • $\begingroup$ This is roughly what I was looking for. In fact, having read your answer makes me think of another explanation: rotations are numerically stable, regardless of determinant of the covariance/correlation matrix. And since PCA can be framed as finding the best rotation of the co-ordinate axis, it will also be numerically stable. $\endgroup$ – probabilityislogic Aug 27 '11 at 23:01
  • $\begingroup$ Yes, for instance in Stan Mulaik's "foundations of factoranalysis" the stability of the pc-rotation (Jacobi-method) was explicitely mentioned, if I recall the source correctly. In my own implementation of factor analysis I do everything after cholesky by rotations: PCA,Varimax, even "principal axis factoring" (PAF in SPSS) can be rebuild on basis of rotations. If the mult regression is based on the cholesky factor L and the part of L which contains the independent variables is in PC-position, then multicollinearity can even better be controlled. $\endgroup$ – Gottfried Helms Aug 28 '11 at 6:29
3
$\begingroup$

PCA is often a means to an ends; leading up to either inputs to a multiple regression or for use in a cluster analysis. I think in your case, you are talking about using the results of a PCA to perform a regression.

In that case, your objective of performing a PCA is to get rid of mulitcollinearity and get orthogonal inputs to a multiple regression, not surprisingly this is called Principal Components Regression. Here, if all your original inputs were orthogonal then doing a PCA would give you another set of orthogonal inputs. Therefore; if you are doing a PCA, one would assume that your inputs have multicollinearity.

Given the above, you would want to do PCA to get a few input variable from a problem that has a number of inputs. To determine how many of those new orthogonal variables you should retain, a scree plot is often used (Johnson & Wichern, 2001, p. 445). If you have a large number of observations, then you could also use the rule of thumb that with $\hat{ \lambda_{i} }$ as the $i^{th}$ largest estimated eigenvalue only use up to and including those values where $\frac{ \hat{ \lambda_{i} } }{p}$ are greater than or equal to one (Johnson & Wichern, 2001, p. 451).

References

Johnson & Wichern (2001). Applied Multivariate Statistical Analysis (6th Edition). Prentice Hall.

$\endgroup$
  • 6
    $\begingroup$ I'm not sure the OP is after PCR. PCA is also a good way to summarize multivariate datasets (not necessarily in order to perform data reduction for subsequent use in a modeling framework), that is approximate the VC matrix to a lower-order one while retaining most of the information. The question seems to be: Am I right when interpreting the first few eigenvalues and PCs (as linear combinations of the original variables) even if there were some collinearity effects? Your response doesn't seem to address directly the OP's question. $\endgroup$ – chl Apr 14 '11 at 11:03
  • 2
    $\begingroup$ good answer about PCA in general, but what about when PCA is the final product? That is, the goal is to output a single PC. @Chl is right on the money with his interpretation of the question $\endgroup$ – probabilityislogic Apr 14 '11 at 12:01
  • $\begingroup$ @chl What is your response to the question: "Am I right when interpreting the first few eigenvalues and PCs even if there were some collinearity effects?" I ask because I am trying to figure out when is it a good idea to keep highly correlated variables when performing dimensionality reduction. Sometimes when we know from theory that two variables are driven by the same latent variables then you should remove one of the variables to not count the effect of the latent variable twice. I am trying to think through when its ok to keep the correlated variables. $\endgroup$ – Amatya Jun 21 '18 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.