2
$\begingroup$

I'm looking to use Spark to calculate PCAs. However I need to get the explained variance for each component and the PCAModel class doesn't appear to provide that.

Is there a way to calculated the explained variance from the component vectors? If so, how would I do this?

If not, can I do this using the SVD? If so, how do I go from the SVD components (U, s, V) to explained variance (and PCA components)?

$\endgroup$
1
  • $\begingroup$ It does now! Use explainedVariance $\endgroup$
    – J Pierret
    Jun 20, 2017 at 15:47

1 Answer 1

3
$\begingroup$

The explained variation is just the eigenvalue corresponding to your eigenvector (the principal component):

https://en.wikipedia.org/wiki/Principal_component_analysis

https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors

It seems the particular software you are using does not automatically provide you with this. If you have an eigenvector and the original matrix (the data), then you just use matrix multiplication to calculate the product of matrix times eigenvector. You will then get a scaled eigenvector, where the scaling factor is the eigenvalue (explained variation) that you are looking for. So there is no need for any other type of decomposition, just matrix multiplication.

If you prefer to do SVD, then the singular values are just the square roots of the eigenvalues, so this would also work.

https://en.wikipedia.org/wiki/Principal_component_analysis#Singular_value_decomposition

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.