Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

I am studying the feature selection in the machine learning field. Lately, I am trying to understand the PCA (Principle Component Analysis). I found a webpage PCA which introduces the PCA and the concept of percentage of variance. However, I am very confused about what "Percentage of Variance" means. I am sorry for my shallow statistics knowledge first. Here are the questions I have

(1) Is there a formal definition or mathematics formula to define "percentage of variance" ?

(2) Do we only use POV for calculating PCA ? Does somewhere else use POV ?

(3) Is POV the same as "Proportion of Variance Explained" ? There are many similar terms online which really confuse me.

Thank you very much,

share|improve this question
Cassie, PCA works with variance-covariance matrix, where variances are its diagonal. Sum of variances is overall multivariate variability. It is this POV is computed of. For more, see here – ttnphns Jul 9 '12 at 6:52
The link you provided is great. Thus, intuitively, when a component has accounted for larger percentage of variance, can I say the component is more important then others ? Moreover, assume I develop a new variable selection method to select only 5 out of 10 original variables. Without using PCA to find eigenvectors, assume I just calculate cov matrix for these 5 variables. Then, can I calculate POV of these 5 variables in the same way ? Or POV can only be applied on eigenvalues ? Thanks, – Cassie Jul 10 '12 at 1:40
Using PCA for feature selection is a little bit different, because the eigenvectors usually do not correspond to input features but rather to a linear combination of these features. Therefore, you can use PCA to reduce the dimensionality of your dataset while still preserving most of the variance by creating a new feature space based on the top $k$ eigenvectors (according to their eigenvalues). However, the "new" features will be different from the original ones. – sebp Jul 10 '12 at 8:17

migrated from cstheory.stackexchange.com Jul 9 '12 at 6:28

1 Answer

up vote 1 down vote accepted

I assume they are referring to the eigenvalues of the corresponding eigenvectors. The eigenvalues in PCA tell you how much variance can be explained by its associated eigenvector. Therefore, the highest eigenvalue indicates the highest variance in the data was observed in the direction of its eigenvector. Accordingly, if you take all eigenvectors together, you can explain all the variance in the data sample. Instead of using the absolute value of variance explained, as indicated by the eigenvalue, you can also get relative numbers by first summing up all eigenvalues and then divide an eigenvalue $\lambda_i$ by this sum $$ \frac{\lambda_i}{\sum_{i=1}^n \lambda_i} . $$ This way you end up with a "percentage of variance" for each eigenvector.

share|improve this answer
Just a clarification: if $X_1,\ldots,X_p$ are the original random variables, then $\sum_{i=1}^p\lambda_i=\sum_{i=1}^p{\rm Var}(X_i)=tr(\Sigma)$ where $\Sigma$ is the covariance matrix. – MånsT Jul 9 '12 at 8:51
Thank you all, these replies explained well. I have added just a small question to clarify further and commented above. Basically, I would like to know if the calculation of POV can be applied on original variables without finding the eigenvalues first. Any idea is welcome. – Cassie Jul 10 '12 at 1:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.