4
$\begingroup$

I am new to PCA and trying to do some analysis on my data set. When I apply PCA to my set of data I get all 100% variance on only one principal component. Does this make any sense?

Any explanation would be appreciated.

$\endgroup$
  • $\begingroup$ Please describe your data: how many variables, what their ranges are, and how you are attempting to do PCA. It's possible you're doing PCA on a covariance matrix and that one variable has a variance so much larger than all the others that it is dominating the calculation. What does the scatterplot matrix of the variables look like? $\endgroup$ – whuber Dec 3 '16 at 5:01
5
$\begingroup$

This means all your variables can be written as a linear transformation of a single one of them, which is a pretty extreme case of linear dependence.

(Why? Because all the principal components of a dataset together are always a basis for the vector space spanned by the original variables.)

$\endgroup$
3
$\begingroup$

To elaborate on @Kodiologist's answer (+1): let's say your data matrix is $X \in \mathbb R^{n \times p}$, and let $X = UDV^T$ be the SVD of $X$. Let's further assume that $X$ has been normalized so that $X^T X = V D^2 V^T$ is the covariance matrix. It is well known that the variance explained by the first $k$ principal components is $\sum_{i=1}^k d_i^2 / \sum_{i=1}^p d_i^2$ (feel free to ask for more details here). If $d_1^2 / \sum_{i=1}^p d_i^2 = 1$ then $d_2 = \dots = d_p = 0$ so this tells us that $X = d_1u_1 v_1^T$, i.e. $X$ is a rank 1 matrix, which is what it means to have all $p$ variables be perfectly collinear. We also have that the covariance matrix is $d_1^2 v_1 v_1^T$ which is also rank 1

$\endgroup$
  • $\begingroup$ Thank very much for your answer. So this means that I have a redundant set of data, so does PCA tell me which one of my variables it is, that all the other variables are linearly dependent on it? $\endgroup$ – mark Dec 3 '16 at 4:15
  • 2
    $\begingroup$ If your matrix is rank 1, then every column is a multiple of any other column. Unless your application tells you otherwise, there isn't really a canonical column that generated the others. $\endgroup$ – jld Dec 3 '16 at 4:24
  • $\begingroup$ MATLAB example: Of such an extreme case: rng(123); svd(repmat(randn(100,1),1,4).*repmat(1:4,100,1)) returns a single non-zero eigenvalue, ie. all the variance lies on a single PC. $\endgroup$ – usεr11852 says Reinstate Monic Dec 3 '16 at 11:52
1
$\begingroup$

One feature with large numeric range might dominate the others.

Try to apply standardization on your dataset before PCA, by removing the mean and scaling to unit variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.