I apologize in advance for the poorness of my statistics and mathematics. I am doing PCA on data (emission spectra) that I know a priori should have two strong components (there are two fluorescent proteins) and a residue; I expect the first two components to be the emission spectra of the two protein, and a residual variance. However, when I run PCA the second component shows a small negative peak corresponding to the first component, as if "maximizing" the first component take away a part of the second. When I try to scatter my data on the first two components, this is reflected by the fact that there is a correlation between the first two components values; I would like to "balance" the first two components, I don't know if this is clear. I guess that maybe PCA is not the best way to perform my analysis, unfortunately I am not familiar with any dimensionality reduction algorithm. which is the simplest similar statistical analysis where I can choose the number of "a priori" components?

This is an example of my emission data

These are the first 2 principal components

This is the scatter plot on the first 2 components


The PDFs (distributions) of the underlying variables or inputs to your PCA are really the key to how you address your question. The most important consideration is scaling which means that 1) they are "continuously" distributed, and 2) that they have been standardized to a mean of zero and std dev of one prior to the PCA. Of course, standardizing is only needed for OLS PCA since it's not a scale-invariant technique and would result in factors dominated by those variables with the largest std deviations. Maximum likelihood PCA is scale-invariant. That said, PCA with categorical or dummy variables is not a good idea. A separate class of dimension reduction techniques for categorical information (e.g., correspondence analysis) has been developed for data of this type.

A few things are always worth reviewing about PCA. First and foremost, it is a highly subjective technique dependent on analyst choice across a myriad of possible model options and specifications. Basically, this means KISS. Next, and in the absence of rotation, PCA produces 1) a mathematically unique solution, 2) orthogonal (right angled, uncorrelated), linear components, 3) Gaussian-like residuals, and 4) the first factor is a kind of "junk" factor (e.g., see Morris Holbrook's -- Columbia Marketing -- papers on FA from the 80s) where many of the input variables will load highly on the first dimension or PC.

Given your description, one immediate option would be to try rotating the PCA solution. There are many ways of doing this, but start with a plain vanilla, orthogonal rotation such as varimax. This would eliminate the uniqueness of the solution, but could provide a better fit in terms of the loadings (or correlations) of the inputs relative to the solution. The usual goal of rotation is to find "simple structure" across the dimensions. The objective of rotation to simple structure is to adjust the loadings in such a way that the inputs load on one dimension and are zero otherwise. Note that rotation does not change the component eigenvalues or "variance explained" across the dimensions. Next, you've indicated that there is strong correlation between the resulting components. This suggests trying an oblique rotation which would relax the constraint that the factors be orthogonal. Promax is one such approach.

Since we know nothing about the variables that you're using as inputs, it could be that taking some transformations prior to doing PCA could improve the solution in the absence of rotation since all PCA is sensitive to outliers. The goal of these transformations would be to further symmetrize any skewness in the residuals output from the untransformed model. Transformations which compress or expand the input PDFs are to be leveraged.

Based on the plethora of 20th c tricks that were developed for outlier identification and deletion, many statisticians would recommend dropping outlying values from the analysis. Personally, I strongly recommend not deleting outliers as they contain valuable information about the behaviors of the inputs, that is, in the absence of ancillary information confirming that these outlying values aren't illegal, fraudulent, erroneous or otherwise "bad" data.

One additional possible avenue is to leverage robust approaches to PCA. These methods are less sensitive to outliers or leverage probabilistic distributions that explicitly model that information. Here are some links to papers that describe them. Both papers have extensive bibliographies for deeper dives. Bear in mind that the cumulative impact from applying any or all of these nontraditional options may completely overturn the resulting solution relative to expectations based on published research using traditional PCA.

Robust Principal Component Analysis? by Candes, Li, Ma and Wright, 2009 Cauchy Principal Component Analysis, Xie and Xing, 2015

  • $\begingroup$ Thank you very much for your kind explanation. I have almost solved my problem with data normalization. Just another question, for the sake of curiosity: I tried to "rotate" the pc matrix, and my output was not exactly what I was expecting: each one of the new pcs had a strong value on only one original dimension, and was almost 0 on each of the other dimensions. Is there a way to perform a similar transformation, giving the constraint that "the first two load should be maximized"? $\endgroup$ – Gardner Jun 14 '15 at 10:23
  • $\begingroup$ @Gardner Are you using the right words? "Dimensions" don't load on dimensions, input variables load on dimensions. If you are saying that, after rotation, the loadings for your input variables were maximized on a single dimension and zero (or close to it) on the other dimensions, then that would be the "simple structure" that is sought when rotating. $\endgroup$ – Mike Hunter Jun 14 '15 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.