I am trying to do PCA using the SVD method manually and compare the results with the results computed by PCA on sklearn.

Let's say that our matrix is A = [[3 0],[0 -2]]

Following the SVD method, I first compute:

1) A^T A = [[9 0], [0 4]] with A^T the transpose matrix of A

2) Then I compute the eigen value given the fact that σ = sqrt(λ) det(A^T A - λ I ) = 0 with I the identity matrix. I found σ1 = 3 and σ2 = 2

3) I can compute now Σ whose value are the σ values.

4) Now I compute the V matrix. To do that I solve : A^T A v = λ v for each λ value. I got : V = [[1 0], [0 1]]

5) To compute U, we can use the formula ui = Avi/σi

U = [ [1 0], [0 -1]]

Then I tried to compute the projection of A (the new k features with k = 2 in this case) using the formula Y = U^T A but the results I got are different from what I got on the pca from sklearn.

My results : Y = [[3 0 ],[0 2]]

sklearn results : [[ 1.80277564e+00 -1.11022302e-16] [-1.80277564e+00 1.11022302e-16]]

Does anyone know where is my mistake ? Does the pca give us different results depending of the method used ?


You should perform the steps above on centered data. You'll need to subtract a vector of each feature's sample mean from your observations.

The steps are as follows.

  1. Compute the covariance matrix $\Sigma$. I assume that your data matrix is observations x features, so this would be $\Sigma = \frac{1}{N}(A-\bar{A})^{T} (A-\bar{A})$.
  2. Compute the eigenvalue decomposition of $\Sigma$. Collect the eigenvectors as the columns of a matrix $V$. Equivalently, you can find the singular value decomposition $\Sigma = V^{T} D V$.

  3. Compute the projection of the (centered) data onto the eigenvectors. This gives the final reduction $\hat{A} = (A-\bar{A}) V$.

Here is my python code. I've used the eigenvector decomposition.

import numpy as np
from sklearn.decomposition import PCA

#use sklearn pca
A=np.array([[3, 0], [0, -2]])
pca = PCA()
print("Sklearn PCA output:")

#And here is by hand.
Cov=np.matmul((A-A.mean(axis=0)).T, (A-A.mean(axis=0)))/A.shape[0]
#Get the eigenvectors from covariance matrix
V = np.linalg.eig(Cov)[1]
#perform the projection
print("Manual PCA output:")
print(np.matmul(A-A.mean(axis=0), V))

And the output.

Sklearn PCA output:
[[-1.80277564e+00 -1.11022302e-16]
 [ 1.80277564e+00  1.11022302e-16]]
Manual PCA output:
[[ 1.80277564  0.        ]
 [-1.80277564  0.        ]]

Note that sklearn is doing the centering behind the scene. You can verify that fitting the PCA on the centered and original data give the same result.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.