For PCA eigenvalues

S = [1.74 1.45 0.93 0.77 0.50 0.30 0.25 0.13 0.00 ...]

where the 2 first PCs explain more than 50% of the variance, is PC1 the most important/significant factor?

Or as the first 2PCs explain 50% of the variance the rest 6 PCs combined are equally significant as the first two? If not, is the PC1 is the most significant part of the variance for the data under investigation?

Essentially, would it make sense to refer to PC1 representation of the data as the most important for the system under investigation?

  • 2
    $\begingroup$ Please tell us first what you mean by "significant" and "important"! The context matters. $\endgroup$
    – whuber
    Jun 17, 2016 at 13:49
  • 1
    $\begingroup$ You keep using the word "scores". I do not think it means what you think it means. $\endgroup$
    – amoeba
    Jun 17, 2016 at 13:49
  • $\begingroup$ @whuber Sure. I have found in the literature that since PCA is a non-parametric method assessing statistical significance can be considered out of the scope, thus I used the word importance. I use PCA as in detection theory for graphs, in other words in finding components/connections according to which a groups of graphs under investigation variate maximally. Hence by important I refer to connections in the graphs that play a crucial role for the network under investigation. Hope that makes sense. $\endgroup$
    – hH1sG0n3
    Jun 17, 2016 at 15:10
  • $\begingroup$ @amoeba by scores I refer to the eigenvalues of the covariance matrix. $\endgroup$
    – hH1sG0n3
    Jun 17, 2016 at 15:11
  • 1
    $\begingroup$ To expand on @amoeba's terminology point, the word score in statistics generally means the gradient of the log likelihood function with respect to some parameter. $\endgroup$ Jun 17, 2016 at 15:29

2 Answers 2

  1. That a variable has high variance does not imply that it's objectively important!
  2. That a variable has low variance does not imply that it's objectively unimportant!

Trivial example: hair color and blood loss in an emergency room

Imagine we have data collected from an emergency room:

  • $x_1$ measures hair color.
  • $x_2$ measures blood loss.

Just for hypothetical purposes, let's say the covariance matrix is: $$\Sigma = \left[ \begin{array}{cc} 100 & 0 \\ 0 & 1\end{array} \right] $$

  • There's tons of variation in hair color $x_1$.
  • On the other hand, blood loss is quite rare. Expected squared deviations from mean blood loss (i.e. the variance) is quite low.

You can instantly see where this is going. An eigenvalue decomposition is:

$$ \left[ \begin{array} {cc} 1 & 0 \\ 0 & 1 \end{array} \right] \left[ \begin{array} {cc} 100 & 0 \\ 0 & 1 \end{array} \right] \left[ \begin{array} {cc} 1 & 0 \\ 0 & 1 \end{array} \right] $$

The first principal component will be $\left[ \begin{array}{c} 1 \\ 0 \end{array} \right]$, the eigenvector associated with the largest eigenvalue.. Hair color describes most of the variation in our data, but that's almost certainly irrelevant in this context!

Disclaimer/note: I have no medical training, and undoubtedly the medical specifics of this hypothetical are all screwed up. Don't take that part literally. The point is about Principal Component Analysis (PCA).

  • $\begingroup$ I agree. But, associating hair colour with blood loss is already a poor choice under probably any context. Conversely, I am implementing PCA in a context that variables' values are of similar scales, already statistically significant and not affected by anything that could be considered noise. $\endgroup$
    – hH1sG0n3
    Jun 21, 2016 at 8:53

The PCA decomposition explains the 'total variance' for your data set. Therfore, it really depends on the question you are asking. If the majority of variance in your data is caused by the effect under investigation then discarding of lower variance PCs can be seen as denoising.

But if your data set has variance caused by something irrelevant for the question you are asking this variance is also covered in PCA and might contribute stronger to the total variance then the effect you are interested in.

Interpreting the loadings of the PCA often helps to find out how many PCs are important for your data set.

Your S looks like the list of eigenvalues corresponding to the PCs, Scores in contrast can be seen as a meassure of how much each PC contributes to each individual object in your data set...

So to answer the question as it is asked - scores do not correlated with the importance of the PC - the eigenvalues can correlate depending on the structure of your data if there are no confounders.

  • $\begingroup$ Thanks very much. I am looking at datasets only relevant to the effect under investigation i.e., weighted connection values among nodes of a graph. Yes, s is the list of the eigenvalues, and I have probably misinterpret those as scores - as scores refer to the score for each object each PC contributes with with regards to variance. As my datasets consist of already statistically significant connection values, I would assume that the reconstruction of the data according to PC1 indicates to the most meaningful less noisy components and so on? Thanks very much, your comment is really helpful. $\endgroup$
    – hH1sG0n3
    Jun 17, 2016 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.