0
$\begingroup$

I was reading about the Principal Component Analysis algorithm. I don't understand why, in order to do dimensionality reduction, we create the covariance matrix and then we extract its eigenvectors.

Compute "covariance matrix":
$$ \Sigma = \frac 1 m \sum_{i=1}^n(x^{(i)})(x^{i)})^T $$ Compute "eigenvectors" of matrix $\Sigma$:
$$ \color{darkblue}{\texttt{[U, S, V]} = \texttt{svd(Sigma);}} $$

After that, why do we select the first k eigenvectors? Why don't we do some ranking of groups of k eigenvectors and then select the best group?

$\endgroup$
  • 4
    $\begingroup$ when they say first k they mean the first k after sorting them from bigger to smaller. they correspond to the eigenvectors that yield the direction of highest variation. $\endgroup$ – Cowboy Trader May 13 '17 at 18:06
  • 5
    $\begingroup$ Just adding a bit of clarity to @CagdasOzgenc. You choose the first k eigenvectors after sorting them by the magnitude of their corresponding eigenvalue. The sum of all these eigenvalues is then the total variance of your dataset, when projected onto the subspace spanned by these k vectors. $\endgroup$ – Matthew Drury May 13 '17 at 18:30
  • $\begingroup$ @MatthewDrury Why is the sum of all eigenvalues the total variance of the dataset? $\endgroup$ – octavian May 13 '17 at 19:49
  • $\begingroup$ @octavian You should be able to find that computation in any good book with an exposition on PCA. $\endgroup$ – Matthew Drury May 13 '17 at 23:39
1
$\begingroup$

The result of transforming the original d -dimensional data onto this new k -dimensional subspace (typically k << d ) is that the first principal component will have the largest possible variance, and all consequent principal components will have the largest possible variance given that they are uncorrelated (orthogonal) to the other principal components.

$\endgroup$
  • 1
    $\begingroup$ You are not connecting the eigenvectors to the principal components as is what the OP was asking about. $\endgroup$ – Michael Chernick May 18 '18 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.