0
$\begingroup$

I always assumed that PCA is exactly the eigendecomposition of the covariance matrix. That is, the explained variances are eigenvalues of the covariance matrix, and components are its eigenvectors. To test my understanding, I have implemented PCA both using the above logic and using an existing library. The results look different. Can somebody explain why? Is there a gap in my understanding, or is it a bug of the code?

nDim = 20
nData = 1000

def naive_pca(d):
    return np.sort(np.linalg.eig(np.cov(d))[0])[::-1]


data = np.random.uniform(0,1,(nDim, nData))

pca = PCA(n_components=nDim)
pca.fit(data)

fig, ax = plt.subplots(ncols=3, figsize=(12,4))
ax[0].plot(pca.explained_variance_)
ax[1].plot(pca.singular_values_)
ax[2].plot(naive_pca(data))
ax[0].set_title("PCA: expained variance")
ax[1].set_title("PCA: singular values")
ax[2].set_title("eigenvalues of covariance")
plt.show()

enter image description here

$\endgroup$
1
  • $\begingroup$ Maybe not exactly what you are looking for, but here is a great explanation of PCA. $\endgroup$ – Stochastic Jun 1 '20 at 9:34
1
$\begingroup$

You messed up with the columns. Scikit-learn assumes that features are in columns, numpy uses rowvar=True as default:

np.random.seed(42)
nDim = 20
nData = 1000

def naive_pca(d):
    return np.sort(np.linalg.eig(np.cov(d))[0])[::-1]

data = np.random.uniform(0,1,(nDim, nData))

pca = PCA(n_components=nDim)
pca.fit(data.T)  # <----- HERE

fig, ax = plt.subplots(ncols=3, figsize=(12,4))
ax[0].plot(pca.explained_variance_)
ax[1].plot(pca.singular_values_)
ax[2].plot(naive_pca(data))
ax[0].set_title("PCA: expained variance")
ax[1].set_title("PCA: singular values")
ax[2].set_title("eigenvalues of covariance")
plt.show()

enter image description here

$\endgroup$
6
  • $\begingroup$ Wait, you are calculating mean over axis=0? That would be the mean over dimensions. That is quite surprising. I expected that for each dimension the mean over datapoints has to be subtracted $\endgroup$ – Aleksejs Fomins Jun 1 '20 at 9:34
  • $\begingroup$ Also, if possible, can you provide formulas for the scaling factors between the three plots $\endgroup$ – Aleksejs Fomins Jun 1 '20 at 9:35
  • $\begingroup$ @AleksejsFomins assuming that you are using scikit learn, axis 0 is samples, axis 1 is dimensions. $\endgroup$ – Tim Jun 1 '20 at 9:36
  • $\begingroup$ Ok, I get it, the problem was that I was plugging the transpose into scikit-learn... I most definitely do not need to subtract the mean however, because np.cov already does that $\endgroup$ – Aleksejs Fomins Jun 1 '20 at 9:41
  • $\begingroup$ @AleksejsFomins yeap. Sorry for the answer that was initially misleading. Also, completely random data is not the best example to try PCA, better try it on meaningful data, that can be reduced to lower dimensionality. $\endgroup$ – Tim Jun 1 '20 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.