1
$\begingroup$

I am bit unsure (Or might be overthinking this) or is the chosen PC somehow applied on to the data, to reduce the dimentionality of the data, or how does one use PC to do any form of mathematical computation?...

I mean each principal component is a linear combinations of the variables.. It basically is an eigenvector and eigenvalue stating the pattern of the data.

Most textbooks seem to end the elaboration after having found the eigenvectors and the eigenvalues, but aren't there another step which implies reducing the data with the PC found?...

$\endgroup$
5
  • $\begingroup$ Most textbooks seem to end the elaboration... Really? Do they ever mention the computation the values of the first m PCs, thereby reducing the dimensionality? $\endgroup$
    – ttnphns
    Commented Jun 19, 2016 at 13:17
  • $\begingroup$ I get how they compute PC's and so on... but aren't the components somehow applied to the original dataset.. each PC is just a linear combinations.. I know that they reduce the dimentionality, but not sure how a PC can reduce the dimentionality of the dataset, as they aren't applied on to the original dataset , only extracted from it. $\endgroup$
    – Sorrow
    Commented Jun 19, 2016 at 13:19
  • $\begingroup$ Principal components can be used to restore the original variables data. $\endgroup$
    – ttnphns
    Commented Jun 19, 2016 at 13:21
  • $\begingroup$ If i used all I would have the complete dataset, thats right, what if you only wanted to have 50% variance. Well one way would be to choose m numbers of PC's such that the requirement is fulfilled, and then what? $\endgroup$
    – Sorrow
    Commented Jun 19, 2016 at 13:24
  • $\begingroup$ Principal components don't reduce dimensionality: if dataset has $n$ features, we can compute $n$ PCs, so dimensionality doesn't change. Dimensionality reduction takes place when we use an arbitrary number of PCs instead of original features in dataset. $\endgroup$
    – Ogurtsov
    Commented Jun 19, 2016 at 15:55

1 Answer 1

2
$\begingroup$

Given you have found the principal modes of variation in your sample (eigenvectors) you use these as your new axis system. When you compute the PC scores you simple project the data on the axial system defined by these eigenvectors.

Let's say your original data is an $N \times p$ matrix $X_0$ with a valid covariance $p \times p$ matrix $C$ (the columns of $X_0$ are assumed to have mean $0$). The covariance $C$ can then be eigen-decomposed as $C = U \Lambda U^T$ where $U$ is the $p \times p$ matrix of eigenvectors (each column is an eigenvector) and $\Lambda$ is the diagonal matrix holding the corresponding eigenvalues. This is the most important step, now $U$ can allow a change of basis from $X_0$ to $Z$, where we define $Z = X_0 U$, $Z$ are now the projected scores $N \times p$ matrix; these are the projections of the original data $X_0$ onto the axis defined by the columns of $U$. Note that here we used the full matrix $U$. If we used the optimal $k$-th dimensional approximation of the data $X_0$ we would use only the first $k$ columns of $U$. As you see the data $X$ are directly employed for their dimensional reduction using the eigenvectors $U$. (Sometimes the columns of $U$ are also called loadings.)

Please also see the thread here; it contains some great answers to assist your intuition further.

Note that I used the work-flow of calculating PCA using the covariance matrix. Most implementations use the singular value decomposition of the original data directly by default because of its better numerical properties in some cases. The results from the two routines are perfectly equivalent. I used the covariance-based approach because I think that it a bit more intuitive. The thread here, contains an excellent answer on how SVD relates to PCA.

$\endgroup$
16
  • $\begingroup$ Isn't there something wrong.. should it not be $Z = U X_0^T$ $\endgroup$
    – Sorrow
    Commented Jun 22, 2016 at 19:08
  • $\begingroup$ Could you elaborate on the term projected scores.. ?? $\endgroup$
    – Sorrow
    Commented Jun 22, 2016 at 19:24
  • $\begingroup$ Err... my bad, sorry typo; I hope it is cleaner now. $\endgroup$
    – usεr11852
    Commented Jun 22, 2016 at 19:32
  • $\begingroup$ Sure no probs, done. $\endgroup$
    – usεr11852
    Commented Jun 22, 2016 at 19:34
  • $\begingroup$ So projection score is basically the loadings? $\endgroup$
    – Sorrow
    Commented Jun 22, 2016 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.