0
$\begingroup$

In PCA, for a given matrix $M_{S\times D}$ where s = samples and d = dimensions, computing covariance matrix of dimension vector and then an eigen decomposition on it leads to eigenvectors which can act as basis for sample vectors. The other way is true as well, covariance matrix of samples --> eigen decomposition --> basis for dimensions. Now going back, if you select only $top k$ of these new basis and project all samples on these selected basis, you have reduced the dimensions. Here, I am not able to connect how operation (covariance) on dimensions leads to basis for samples which ultimately leads to dimensionality reduction.

Another case is when $samples < dimensions$, the shape of the covariance matrix is $d\times d$ matrix, but when we compute eigenvectors, only $samples-1$ vectors make sense and have significant eigenvalues. (I understand this logic, thanks to another good answers on cross validated) But what I am not able to understand is even if we had $samples > dimensions$, the shape of covariance matrix would have been the same i.e. $d\times d$, then why do we find more eigenvectors with values greater than 0? What is the covariance matrix really representing here?

One quick example of this using python and sklearn PCA code is below,

Code

# create data and call PCA
from sklearn.decomposition import PCA
rng = np.random.RandomState(123)
X = rng.randn(sample, dimension)
pca = PCA()
pca.fit(X)

# print
print("Components: ", pca.components_)
print("Variance: ", pca.explained_variance_)

Case 1: When $sample<dimension$, to be exact (sample=4, dimension=6)

>>
Components: [[ 0.73687429 -0.08513232 -0.2547823   0.21629604  0.4724457   0.33891861]
 [ 0.29805133  0.14415459  0.20833423  0.72175483 -0.55190314 -0.14647387]
 [ 0.11062469  0.68912637 -0.21008962 -0.30710435 -0.37043902  0.48702282]
 [-0.22282547  0.53006845 -0.4964642   0.30991411  0.35494731 -0.44817986]]
Variance: [5.73236908e+00 2.50133823e+00 1.00408590e+00 8.06318828e-32]

Case 1: When $sample>dimension$, to be exact (sample=7, dimension=6)

>>
Components:  [[ 0.60370803 -0.25109471 -0.33005044  0.15333665  0.50029574  0.43559938]
 [-0.28232164 -0.65222972 -0.3481593  -0.32944686  0.26763265 -0.43990172]
 [ 0.37070419  0.05233376 -0.32561631  0.5450405  -0.25965749 -0.62395709]
 [-0.0840187  -0.64147911  0.53590501  0.51373007 -0.12764523  0.11848767]
 [ 0.29570739  0.15881516  0.59223915 -0.12690706  0.55441616 -0.46163297]
 [-0.56912873  0.26849826 -0.1608864   0.53931392  0.5357092   0.01651937]]
Variance:  [3.62940095 2.6120547  1.87300687 0.59170826 0.31615464 0.05475089]

I am just getting started with PCA, sorry if the question is quite basic. Thank you!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.