PCA: Out of Sample Variance Explained [closed]

The eigenvalues of a PCA model can be used to calculate the variance explained of each principal component.

I'm curious how one might train a PCA model on a subset of data, and then use the eigenvectors of that model to calculate the variance explained on unseen (out-of-sample/ test) data.

Any suggestions for python or R implementations would be highly appreciated.

closed as off-topic by Michael Chernick, kjetil b halvorsen, Peter Flom♦Jan 16 '18 at 12:08

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, kjetil b halvorsen, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

• Asking for Python or R implementations make this possibly off-topic (voting to leave open since the actual question seems to be statistical, but you may want to remove the last sentence) – Juho Kokkala Jan 16 '18 at 7:30

Unless your train and test data follow the exact same distribution, the eigenvectors of your train data are different from test.

Let's assume that train and test distributions are very similar and we can use the same principle components. If X_train and X_test are two pxn and pxm. (n and m are number of samples).

import numpy as np
X_train = X_train - np.mean(X_train,axis=1)[:,np.newaxis]
X_test  = X_test  - np.mean(X_test, axis=1)[:,np.newaxis]
Sigma_train = np.dot(X_train,X_train.T)/n
V,U = np.linalg.eigh(Sigma_train)


Using the assumption mentioned above, you can calculate the projections of your data:

Y_test = np.dot(U.T,X_test)


The variance of each row is the test variance along the principle components.

Y_var = np.sum(Y_test**2,axis=1)/m


However, PCA is unsupervised, so in practice we don't have to use train eigenvectors for test data.

Note: There's a concept called robust PCA for when test and/or train contain gross outliers. But I don't think you'd be interested in that, especially since those methods don't find the maximum variance.

• Thank you very much for the excellent explanation! Just to clarify, beyond maintaining similar distributions, are we also making any implicit assumptions about the covariance structure? Furthermore, in order to test the stability of two eigenvectors across samples, would it be appropriate to take the dot product of them? Thanks again. – abstract Jan 16 '18 at 0:25
• Re your first question: Yes, by distribution I meant the covariance matrix that specifies the Gaussian distribution. Because, in standard PCA the distribution is always Gaussian. Re your second question: As long as the order of the eigenvalues (in terms of size) is the same between train and test, the eigenvectors should ideally be parallel. However, if the maximum variance directions are different, there's no way to tell which eigenvectors to compare against each other. – idnavid Jan 16 '18 at 2:00
• This is not correct. You're subtracting the mean of the out-of-sample data. But, the mapping PCA defines from the high dimensional to the low dimensional space requires subtracting the mean of the training data. This also implies that you can't just add up the variance of the projections of the out-of-sample data. – user20160 Jan 16 '18 at 8:30