1
$\begingroup$

I've been reading some documentation about PCA and trying to use scikit-learn to implement it. But I struggle to understand what are the attributes returned by sklearn.decompositon.PCA From what I read here and the name of this attribute my first guess would be that the attribute .components_ is the matrix of principal components, meaning if we have data set X which can be decomposed using SVD as

X = USV^T

then I would expect the attribute .components_ to be equal to

XV = US.

To clarify this I took the first example of the wikipedia page of Singular Value Decomposition (here), and try to implement it to see if I obtain what is expected. But I get something different. To be sure I didn't make a mistake I used scipy.linalg.svd to do the Singular Value Decomposition on my matrix X, and I obtained the result described on wikipedia:

X = np.array([[1, 0, 0, 0, 2],
          [0, 0, 3, 0, 0],
          [0, 0, 0, 0, 0],
          [0, 2, 0, 0, 0]])
U, s, Vh = svd(X)
print('U = %s'% U)
print('Vh = %s'% Vh)
print('s = %s'% s)

output:

U = [[ 0.  1.  0.  0.]
[ 1.  0.  0.  0.]
[ 0.  0.  0. -1.]
[ 0.  0.  1.  0.]]
Vh = [[-0.          0.          1.          0.          0.        ]
[ 0.4472136   0.          0.          0.          0.89442719]
[-0.          1.          0.          0.          0.        ]
[ 0.          0.          0.          1.          0.        ]
[-0.89442719  0.          0.          0.          0.4472136 ]]
s = [ 3.          2.23606798  2.          0.        ]

But with sk-learn I obtain this:

pca = PCA(svd_solver='auto', whiten=True)
pca.fit(X)
print(pca.components_)
print(pca.singular_values_)

and the output is

[[ -1.47295237e-01  -2.15005028e-01   9.19398392e-01  -0.00000000e+00
-2.94590475e-01]
[  3.31294578e-01  -6.62589156e-01   1.10431526e-01   0.00000000e+00
6.62589156e-01]
[ -2.61816759e-01  -7.17459719e-01  -3.77506920e-01   0.00000000e+00
-5.23633519e-01]
[  8.94427191e-01  -2.92048264e-16  -7.93318415e-17   0.00000000e+00
-4.47213595e-01]]
[  2.77516885e+00   2.12132034e+00   1.13949018e+00   1.69395499e-16]

which is not equal to SV^T (I spare you the matrix multiplication, since anyway you can see that the singular values are different from the one obtained above). I tried to see what happened if I set the parameter withened to False or the parameter svd_solver to 'full' but it doesn't change the result.

Do you see a mistake somewhere, or do you have an explanation?


$\endgroup$
  • $\begingroup$ See stats.stackexchange.com/questions/134282/… $\endgroup$ – Tim Nov 4 '17 at 23:02
  • 1
    $\begingroup$ PCA = SVD after centering. You did not center. Hence the difference. $\endgroup$ – amoeba says Reinstate Monica Nov 4 '17 at 23:33
  • 1
    $\begingroup$ Thanks a lot amoeba. Indeed, I didn't pay attention to this centering step. Have a nice day! $\endgroup$ – GWa Nov 19 '17 at 9:04
1
$\begingroup$

Annoyingly there is no SKLearn documentation for this attribute, beyond the general description of the PCA method.

Here is a useful application of pca.components_ in a classic facial-recognition project (using data bundled with SKL, so you don't have to download anything extra). Working through this concise notebook is the best way to get a feel for the definition & application of pca.components_

From that project, and this answer over on StackOverflow, we can learn that pca.components_ is the set of all eigenvectors (aka loadings) for your projection space (one eigenvector for each principal component). Once you have the eigenvectors using pca.components_, here's how to get eigenvalues.

For further info on the definitions & applications of eigenvectors vs loadings (including the equation that links all three concepts), see here.

For a 2nd project/notebook applying pca.components_ to (the same) facial recognition data, see here. It features a more traditional scree plot than the first project cited above.

$\endgroup$
  • 1
    $\begingroup$ This introduces a lot of confusion. The link you gave DOES contain documentation for this attribute. And my old comment above explained what OP did wrong. $\endgroup$ – amoeba says Reinstate Monica Apr 26 '18 at 11:36
0
$\begingroup$

pca.components_ is not other than the Loading Scores. With PCA with SVD (singular value decomposition), the principal components are scaled to 1. Imagine the loading scores as a recipe of a cocktail where our PC1 is made with (loading scores for PC1) 0.97 part of Gen1 and 0.242 parts of Gen2, and PC2 is made -0.242 parts of Gen1 and 0.97 of Gen2 (loading scores for PC2).

This is indeed giving us the called Singlular Vector or eigenvector for each component ( the loading scores are the coefficients of each variable for the first component versus the coefficients for the second component).

This also indicates which variables have the largest effect on each component. Loadings can range from -1 to 1. Loadings close to -1 or 1 indicate that the variable strongly influences the component. Loadings close to 0 indicate that the variable has a weak influence on the component. Evaluating the loadings can also help you characterize each component in terms of the variables.

I would advice to watch the flowing video: https://www.youtube.com/watch?v=FgakZw6K1QQ

$\endgroup$
0
$\begingroup$

I hope the following code helps. What you need is just a little "np.transpose"!

pca_math1 = np.matmul(original_matrix, (pca_obj.components_))
pca_math2 = np.matmul(original_matrix, np.transpose(pca_obj.components_))
pca_sklearn = pca_obj.transform(original_matrix)

print((pca_sklearn - pca_math1)[0][0]) # You can check for other elements as well.
# >>> some number which is NOT "0.0".
print((pca_sklearn - pca_math2)[0][0]) # Again, you can check for other elements as well.
# >>> 0.0

Note:

np.allclose(pca_sklearn, pca_math2)
# >>> True
np.array_equal(pca_sklearn, pca_math2)
# >>> False
$\endgroup$
  • 1
    $\begingroup$ Although implementation is often mixed with substantive content in questions, we are supposed to be a site for providing information about statistics, machine learning, etc., not code. It can be good to provide code as well, but please elaborate your substantive answer in text for people who don't read this language well enough to recognize & extract the answer from the code. $\endgroup$ – gung - Reinstate Monica Jul 31 '19 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.