Timeline for Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?
Current License: CC BY-SA 3.0
24 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 15, 2020 at 22:05 | answer | added | wcochran | timeline score: 0 | |
| Apr 28, 2020 at 16:45 | answer | added | nicrie | timeline score: 3 | |
| Mar 20, 2020 at 15:28 | answer | added | Hamed | timeline score: 0 | |
| Jun 8, 2019 at 3:20 | comment | added | Jacob |
Sometimes I get different decompositions. E.g. C = [ 0.236553 -0.020460 0.029987;-0.020460 0.366393 0.018122;0.029987 0.018122 0.330042] and in Octave, I get a reflection with [Ve,D] = eig(C) and a rotation with [U,S,V] = svd(C). How do you deal with that when I need a rotation for visualizing the confidence ellipsoid?
|
|
| Feb 16, 2018 at 7:31 | answer | added | Gruff | timeline score: 2 | |
| Nov 23, 2017 at 14:49 | answer | added | Mosalx | timeline score: 10 | |
| S Nov 17, 2017 at 18:18 | history | suggested | Rodrigo de Azevedo |
Edited tags
|
|
| Nov 17, 2017 at 17:30 | review | Suggested edits | |||
| S Nov 17, 2017 at 18:18 | |||||
| Nov 17, 2017 at 10:55 | comment | added | Federico Poloni | @broncoAbierto Interesting -- I didn't know about these details, in particular that two different algorithms are used for these two tasks in Lapack currently. | |
| Nov 16, 2017 at 21:31 | history | edited | amoeba | CC BY-SA 3.0 |
deleted 145 characters in body
|
| Nov 16, 2017 at 19:22 | comment | added | gunakkoc | As @amoeba mentioned in comments, the question is not about comparing SVD on data to eigen decomposition on covariance matrix, it is about why svd was used on "covariance" matrix. | |
| Nov 16, 2017 at 19:19 | comment | added | Vladislavs Dovgalecs | TL;DR The answer is in your question: SVD is much more numerically stable than EIG | |
| Nov 16, 2017 at 19:14 | history | tweeted | twitter.com/StackStats/status/931238944418693120 | ||
| Nov 16, 2017 at 17:01 | answer | added | Aksakal | timeline score: 15 | |
| Nov 16, 2017 at 16:24 | comment | added | cangrejo | @FedericoPoloni I found a relevant paper. See my updated answer if you're interested. | |
| Nov 16, 2017 at 15:46 | comment | added | cangrejo | @FedericoPoloni Actually, I think A. Ng might be right. Apparently, the LAPACK implementation of the SVD uses a divide-and-conquer approach as described here: en.wikipedia.org/wiki/Divide-and-conquer_eigenvalue_algorithm as opposed to the conventional QR algorithm used by the eigendecomposition code. The former appears to be more stable, although I have not read a detailed account of why. | |
| Nov 16, 2017 at 14:28 | comment | added | Federico Poloni |
That might be based on an incorrect understanding: doing an SVD of the data matrix is more stable than using eig or svd on the covariance matrix, but as far as I know there is no big difference between using eig or svd on the covariance matrix --- they are both backward stable algorithms. If anything, I would put my money on eig being more stable, since it does fewer computations (assuming both are implemented with state-of-the-art algorithms).
|
|
| Nov 16, 2017 at 11:56 | answer | added | cangrejo | timeline score: 27 | |
| Nov 16, 2017 at 11:37 | comment | added | amoeba |
In Matlab x=randn(10000); x=x'*x; tic; eig(x); toc; tic; svd(x); toc; on my machine outputs 12s for eig() and 26s for svd(). If it's so much slower, it must at least be more stable! :-)
|
|
| Nov 16, 2017 at 11:22 | comment | added | amoeba | Some information about this here: de.mathworks.com/matlabcentral/newsreader/view_thread/21268. But note that any explanation about why one algorithm would be more stable than another is going to be very technical. | |
| Nov 16, 2017 at 11:19 | history | edited | amoeba | CC BY-SA 3.0 |
edited title; edited tags
|
| Nov 16, 2017 at 11:18 | comment | added | amoeba | I mean they are mathematically the same. Numerically they might indeed use different algorithms and one might be more stable than another (as Ng says). This would be interesting to know more about, +1. | |
| Nov 16, 2017 at 10:52 | comment | added | amoeba | For square symmetric positive semidefinite matrix (such as covariance matrix), eigenvalue and singular value decompositions are exactly the same. | |
| Nov 16, 2017 at 10:11 | history | asked | DongukJu | CC BY-SA 3.0 |