# When doing dimensionality reduction with PCA, why do we take k eigenvectors?

I was reading about the Principal Component Analysis algorithm. I don't understand why, in order to do dimensionality reduction, we create the covariance matrix and then we extract its eigenvectors.

Compute "covariance matrix":
$$\Sigma = \frac 1 m \sum_{i=1}^n(x^{(i)})(x^{i)})^T$$ Compute "eigenvectors" of matrix $\Sigma$:
$$\color{darkblue}{\texttt{[U, S, V]} = \texttt{svd(Sigma);}}$$

After that, why do we select the first k eigenvectors? Why don't we do some ranking of groups of k eigenvectors and then select the best group?

• when they say first k they mean the first k after sorting them from bigger to smaller. they correspond to the eigenvectors that yield the direction of highest variation. Commented May 13, 2017 at 18:06
• Just adding a bit of clarity to @CagdasOzgenc. You choose the first k eigenvectors after sorting them by the magnitude of their corresponding eigenvalue. The sum of all these eigenvalues is then the total variance of your dataset, when projected onto the subspace spanned by these k vectors. Commented May 13, 2017 at 18:30
• @MatthewDrury Why is the sum of all eigenvalues the total variance of the dataset?
– bsky
Commented May 13, 2017 at 19:49
• @octavian You should be able to find that computation in any good book with an exposition on PCA. Commented May 13, 2017 at 23:39