I have posted an almost similar question last week but I want to get some more of the theoretical background Difference in Variance of factor scores for supplementary and active observations PCA.

I am doing PCA on genetic data (number of variables >> number of rows) and all data points take values in {0,1,2}. The aim of doing the PCA is to isolate the effect of different origins (People from Europe have different genetic structure than people from Asia).

I have run the PCA with several different packages on two kinds of different data sets. One with little structure ( all the samples had the same origin) and one including samples from two different world regions.

I use a subset of the data to find PCs and then apply the loading matrix to supplementary samples. What I observe is the following. In the case of the data set with little structure, as expected the eigenvalues are almost similar and so the proportion of total population variance explained by the first principal component is small. Now if I apply the loadings matrix to supplementary individuals that have been standardized using the mean of the active sample the score variance of these supplementary individuals is much smaller (3 to 5 times smaller). I then did the PCA only on the supplementary individuals and it turns out that the PC1 of the two subsets are poorly correlated (~ $0.07$).

When I do the exact same thing on data including samples from two different world regions than the difference in Score Variance between the active and supplementary individuals is much smaller ( $\frac{Var(Score_{passive})}{Var(Score_{active})} = 0.9$) and the PC1 of the two subset are strongly correlated (~ $0.95$). However the the Score Variance of the passive sample is still smaller than the one of the active sample.

Now my question:

Is this a global result? Is the Score Variance of the supplementary individuals always smaller than the one of the active? I'd really like to understand but I'm having trouble getting the mathematics and Intuition behind that straight.


1 Answer 1


(This is largely a repetition of your previous question, maybe you should have edited that one instead. But anyway.)

(1) Your datasets seem to be very similar, consist of genetic data and differ only by the regions where the subjects came from. Therefore I guess one can assume that the variance of individual dimensions (loci?) is pretty similar in each dataset, the number of dimensions is the same, and so the total variance is also very similar.

(2) First eigenvalues of covariance matrix of your "active" subset in dataset 1 (same regions) are very similar to each other. As the datasets are also similar, I guess this means that the same is true for the "supplementary" subset of dataset 1 and also for both subsets of dataset 2 (different regions).

(3) In the dataset 2 the PC1 of the "active" subset" and the PC1 of the "supplementary" subset are very highly correlated. This means that it is essentially the same component. The reason PC1_active explains a little bit less variance in the "supplementary" subset than in the "active" subset is simply because this PC is optimized for the active subset. The subsets have slightly different PCs, so no wonder a PC from another subset would explain less variance.

(4) In this sense, yes, it is a global result. However, these comparisons only make sense if the total variance is the same between subsets (which I assume is true based on (1)). In general it makes more sense to compare the variance of the supplementary data projected onto the PC1_active to the variance of the supplementary data (same data) projected onto the PC1_passive (different PC), instead of comparing it to the variance of the active data (different data) projected onto the PC1_active (same PC).

(4) What happens in your dataset 1, I don't know. If (1) and (2) is true, then you should get similar explained variance with both PCs. You get 3-5 times difference, and it is weird. You need to explore this in more detail. I would compute the amount of variance first 5 active PCs explain in the active dataset and passive dataset, and the amount of variance first 5 passive PCs explain in the active and passive dataset (it's usually more convenient to deal with relative variances, i.e. normalized by the total variance, which is sum of all eigenvalues). With these 20 numbers you can maybe better see what is going on.

Finally, I am not sure what you are trying to achieve. If your goal is to "isolate the effect of different origins" (so you suppose that the effect is localised in loci?), then PCA would not help you much. If your goal is to classify subjects by the origin, then you would be better off doing dimensionality reduction with PLS (partial least squares) instead of PCA, see

  • $\begingroup$ Thank you so much for your answer. I am sorry I wasn't very precise on what I am trying to achieve. I want to control for population stratification in my regression model for a genome association study. I.e. I would like to include the first PCA (that represents origin) in the regression. My dataset is gonna bu huge, so I wanted to save computation time and calculate them only for a subsample. I am looking for ways to efficiently compute PCAs for a huge sample (90k x 400k). Thanks a lot for your helo that helped me a lot to get the intuitions right. $\endgroup$ Aug 13, 2013 at 10:15
  • $\begingroup$ @JohannesEbert: three points come to mind. First, I would really recommend using the first PLS component rather than the first PCA component in your regression (or maybe use PLS regression directly). Second, in either case you will be much better off with an iterative method to find the first PC rather than using the standard eigenvalue method; try iterative methods that do not precompute covariance matrix (maybe R can do it for you, this I don't know). Third, if you do want/need to use subsamples, try selecting them randomly (and use several random subsamples, hoping to find the "same" PC). $\endgroup$
    – amoeba
    Aug 14, 2013 at 22:00
  • $\begingroup$ Thank you, I didn't know about iterative methods before. I will trz to do randomized SVD and do it on the whole sample. $\endgroup$ Aug 15, 2013 at 8:55
  • $\begingroup$ For iterative methods google "Power iteration" and also "Lanczos algorithm"; for these algorithms however you will need to compute covariance matrix first, and for huge datasets this can be very costly by itself. Look for "probabilistic PCA" approach, which does not require precomputing covariance matrix. In its simplest form you can even find it on the wiki: en.wikipedia.org/wiki/… $\endgroup$
    – amoeba
    Aug 15, 2013 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.