Timeline for Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?
Current License: CC BY-SA 4.0
22 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| S Jan 7, 2020 at 15:46 | history | suggested | Kevin Zakka | CC BY-SA 4.0 |
removed duplicate word
|
| Jan 7, 2020 at 11:13 | review | Suggested edits | |||
| S Jan 7, 2020 at 15:46 | |||||
| Nov 19, 2017 at 1:22 | comment | added | Aksakal | @FedericoPoloni, on FP arithmetic and not knowing the exact answer I disagree. In this case I do know the answer with enough precision for this task. On 2x2 you have a fair point. I'll think of something. | |
| Nov 18, 2017 at 17:39 | comment | added | Federico Poloni |
And another problem is that both svd and eig probably take ad-hoc code paths for 2x2 problems (using formulas that will not be much different from your "exact" one). Their performance on a 2x2 problem may not reflect that obtained on a larger matrix.
|
|
| Nov 18, 2017 at 17:38 | comment | added | Federico Poloni | I'm not sure what that formula is, but it is evaluated in the same floating point arithmetic that is used for svd and eig, so it will have errors, too. You should compare against eigenvalues computed with higher accuracy, otherwise you will never know what the true eigenvalues of C are. | |
| Nov 18, 2017 at 17:04 | comment | added | Aksakal | I modified the code to calculate the exact eigen value | |
| Nov 18, 2017 at 17:04 | history | edited | Aksakal | CC BY-SA 3.0 |
actual eigen
|
| Nov 18, 2017 at 14:39 | history | edited | Aksakal | CC BY-SA 3.0 |
error explanation
|
| Nov 18, 2017 at 9:32 | comment | added | Federico Poloni |
And there is a more fundamental issue. Even if those were the true eigenvalues of the covariance matrix $C$, the matrix you pass to eig or svd is not the exact matrix $C$, but a computed approximation to it, which might as well be singular for all we know: damage has already been done in the matrix product. I think the only meaningful way to assess accuracy is comparing the eigenvalues of C with the same eigenvalues computed with many more significant digits. If you have access to Sagemath, you can do it with RR = RealField(prec=300); CC = matrix(C).change_ring(RR); CC.eigenvalues().
|
|
| Nov 18, 2017 at 9:28 | comment | added | Federico Poloni | Also, I don't see why those should be the true eigenvalues. | |
| Nov 18, 2017 at 9:26 | comment | added | Federico Poloni |
That code doesn't run --- there is an esp typo.
|
|
| Nov 17, 2017 at 17:00 | history | edited | Aksakal | CC BY-SA 3.0 |
accuracy in %
|
| Nov 17, 2017 at 10:51 | comment | added | Federico Poloni |
Also, I don't think one 2x2 example is sufficient. For instance, if you simply change that np.random.seed(1) to np.random.seed(2), then SVD returns 9.81307787e-18 as second eigenvalue and the eigendecomposition returns 1.38777878e-17, which seems to be a lot more accurate (it coincides with the result computed with Sagemath and 300 significant digits).
|
|
| Nov 17, 2017 at 10:44 | comment | added | Federico Poloni | Why do you think the eigendecomposition "fails"? Both results seems to be accurate within machine precision * norm(C). | |
| Nov 16, 2017 at 22:27 | comment | added | Aksakal | @ttnphns, non positive definite matrix is an issue, of course | |
| Nov 16, 2017 at 22:03 | comment | added | ttnphns | This is a good answer. I wish, to mention, however, that svd can't detect negative eigenvalues when there are any and you want to see them (if covariance matrix is not original but is, say, smoothed or estimated somehow or inferred or comes out of pairwise deletion of missing values). Moreover, eig on cov matrix remains a little bit faster than svd on it. | |
| Nov 16, 2017 at 21:02 | history | edited | amoeba | CC BY-SA 3.0 |
added 23 characters in body
|
| Nov 16, 2017 at 19:16 | history | edited | Aksakal | CC BY-SA 3.0 |
eigen clarification
|
| Nov 16, 2017 at 18:56 | comment | added | Aksakal | @amoeba, I clarified the relation of SVD and PCA | |
| Nov 16, 2017 at 18:55 | history | edited | Aksakal | CC BY-SA 3.0 |
SVD to eigen
|
| Nov 16, 2017 at 18:24 | comment | added | amoeba | Yes, but here OP is asking about SVD vs EIG applied both to the covariance matrix. | |
| Nov 16, 2017 at 17:01 | history | answered | Aksakal | CC BY-SA 3.0 |