I'm taking Andrew NG's Machine Learning Course and got to the part of Principal Component Analysis. Andrew's implementation of PCA aroused 2 questions for me.

1. Let's say that we have the data matrix $X$, which is $m$ x $n$, with the $m$ rows representing the number of samples and the $n$ columns are the number of the measurements and assume that mean normalization and feature scaling has already been done. Now to do PCA I need the eigenvectors and eigenvalues of the covariance matrix. The covariance matrix of $X$ is defined as $C_X = \dfrac{1}{m - 1}X^TX$. So if I define another matrix, say $Y$ to be $Y = \dfrac{1}{\sqrt{m - 1}} X$ and therefore $Y^TY = \dfrac{1}{m - 1}X^TX = C_X$ and do SVD on this matrix $Y$, aiming for $Y = U \Sigma V^T$, I will get in $V$ the eigenvectors of $Y^TY$, in $U$ the eigenvectors of $YY^T$ and $\Sigma^2$ will contain the eigenvalues of either of them, $YY^T$ or $Y^TY$. So we can conclude that the matrix $V$ contains the eigenvectors of the covariance matrix and $\Sigma ^2$ contains the eigenvalues of the covariance matrix. Therefore it is this matrix $V$ that I should use to project my data matrix $X$ with something like $Z = XV$. This is what I have seen from multiple other sources, including many questions from this website (for example in this answer). But when I watched Andrew NG's algorithm he used the matrix $U$ instead of $V$ and projected the data on the PC's doing something like $Z=U^T X$. This seems very odd to me since pretty much all other sources use $V$, but he used $U$. How can this be correct?

2. Again, virtually all other sources that I have checked use either SVD on the data matrix $X$ or eigendecomposition of the covariance matrix $C_X = \dfrac{1}{m - 1}X^T X$, in order to get the eigenvectors and eigenvalues of the covariance matrix, since that is what we need for PCA. But again Andrew NG does something different. He does SVD on the covariance matrix $C_X$. Again this seemed very odd to say the least but then I stumbled upon this answer which argues that this computation gives the exact same results. From that answer I understood why the eigenvectors are the same, but why the eigenvalues are the same remained a mistery to me. Shouldn't SVD on the covariance matrix give us the eigenvalues squared? And if we have the eigenvalues squared, won't that ruin our ratios when we are trying to choose the number of principal components to keep. For example if we wanted to choose the number of PC's such that we have 95% of the total variance retained, won't the fact that we have the eigenvalues squared ruin our ratios? So shouldn't we take the square root of these values to get the right answer? The person who answered gave a mathematical proof that they are in fact the same, but I couldn't understand this proof. Specifically, he said:

The singular value decomposition of a matrix $A$ is $A=UΣV^T$, where the columns of $V$ are eigenvectors of $A^T A$ and the diagonal entries of $Σ$ are the square roots of its eigenvalues, i.e. $\sigma_{ii} = \sqrt{\lambda_i(A^T A)}$.

Now why is that? Where does that $A^T A$ come from? Shouldn't it be that $\sigma_{ii} = \sqrt{\lambda_{ii}}$ ? He kept multiplying the $\sigma$'s with that weird $A^TA$ in the rest of the proof but I just don't understand where does that multiplication with this matrix come from?

I'm sorry if this seems like a really long question, but I've been thinking about these for quite a while now and I just couldn't find/come up with an answer to any of these.

  • $\begingroup$ Regarding 1: Define $B=\frac{1}{\sqrt{m-1}} X^\top$. Now we have $C_B = B^\top B = \frac{1}{m-1}XX^\top$. What happens when you apply SVD to $B$? $\endgroup$ – Sycorax Aug 27 '19 at 17:20
  • $\begingroup$ @Sycorax If I do that it seems to me that indeed, I would use $U$ instead of $V$ for projecting. But it looks like for this to be the case I would have to use SVD on the transpose of the data matrix $X$ because of $B$'s definition. However that was not used in the course. SVD was used on the matrix $X$ itself, it was not transposed prior to applying SVD and the matrix $U$ was still the one used. This is why I'm confused. $\endgroup$ – user257444 Aug 27 '19 at 20:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy