I know for PCA, it's true that the first N eigenvectors have N greatest variance.

But I'm not sure whether that's also true for NMF(Non-negative Matrix Factorization). For example, this method(Standard Nonnegative Matrix Factorization (NMF) [Lee2001], [Lee1999].): http://nimfa.biolab.si/nimfa.methods.factorization.nmf.html

And are there articles that write about how to calculate the variance of first N eigenvectors? What is the best k(k means counts of eigenvector) for dimension reduction?



Firegun - NMF and PCA are very different in theory, although their models and results are notationally v/similar (i.e., X = AB). NMF aims to decompose X to sparse factors that can explain it (while the data and factors are all positive), whereas PCA does the decomposition based on the amount of variance that a given factor/PC can explain. Thus, in the latter, variance-sorting is a sensible way of scoring components, whereas in NMF, it might not necessarily be so ... If you would like to use NMF for dimensionality reduction, the best way might be to look at the factors and see which ones are irrelevant to your signal (say, are noise) and zero their contribution (again, you still can do variance scoring, but if you think that variance scoring will define the importance, you might as well use PCA).


Actually in PCA, there is no order on the eigenvectors and eigenvalues. They are just eigenvectors and their associated eigenvalues. It is a best practise to sort them bei their eigenvalues, as experience shows the eigenvectors associated with strong eigenvalues are more important and/or significant. However, this is just a rule of thumb. Fact s that the Eigenvalues measure the variance in the one-dimensional subspace.

If you find a good other measure, you can sort them differently. For example, you might want to compute the variance only on the .10-.90 quantile to avoid outliers from influencing the result too much.

And if you get vectors out of something else, you can compute the variance of the projection to this 1d space, and you get a value comparable to the eigenvalues in the PCA case.

  • 2
    $\begingroup$ Hi, Anony-Mousse. I think the first sentence in this answer could cause some confusion since PCA can be formulated as the solution to a couple of different (sequential) optimization problems. In that sense, the eigenvalues and corresponding eigenvectors are very much ordered. There is, particularly conceptually, a significant difference between PCA and simply finding the spectral decomposition of a covariance matrix. $\endgroup$ – cardinal Nov 8 '12 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.