1
$\begingroup$

I was reading about the Principal Component Analysis algorithm. I don't understand why, in order to do dimensionality reduction, we create the covariance matrix and then we extract its eigenvectors.

Compute "covariance matrix":
$$ \Sigma = \frac 1 m \sum_{i=1}^n(x^{(i)})(x^{i)})^T $$ Compute "eigenvectors" of matrix $\Sigma$:
$$ \color{darkblue}{\texttt{[U, S, V]} = \texttt{svd(Sigma);}} $$

After that, why do we select the first k eigenvectors? Why don't we do some ranking of groups of k eigenvectors and then select the best group?

Improve this question
$\endgroup$
4
  • 4
    $\begingroup$ when they say first k they mean the first k after sorting them from bigger to smaller. they correspond to the eigenvectors that yield the direction of highest variation. $\endgroup$
    – Cagdas Ozgenc
    May 13, 2017 at 18:06
  • 5
    $\begingroup$ Just adding a bit of clarity to @CagdasOzgenc. You choose the first k eigenvectors after sorting them by the magnitude of their corresponding eigenvalue. The sum of all these eigenvalues is then the total variance of your dataset, when projected onto the subspace spanned by these k vectors. $\endgroup$
    – Matthew Drury
    May 13, 2017 at 18:30
  • $\begingroup$ @MatthewDrury Why is the sum of all eigenvalues the total variance of the dataset? $\endgroup$
    – bsky
    May 13, 2017 at 19:49
  • $\begingroup$ @octavian You should be able to find that computation in any good book with an exposition on PCA. $\endgroup$
    – Matthew Drury
    May 13, 2017 at 23:39

1 Answer 1

Reset to default
1
$\begingroup$

The result of transforming the original d -dimensional data onto this new k -dimensional subspace (typically k << d ) is that the first principal component will have the largest possible variance, and all consequent principal components will have the largest possible variance given that they are uncorrelated (orthogonal) to the other principal components.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You are not connecting the eigenvectors to the principal components as is what the OP was asking about. $\endgroup$
    – Michael R. Chernick
    May 18, 2018 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.