1
$\begingroup$

I have the following experience about PCA and I don't understand why. I first do PCA on the original dataset, say, a collection of three-dimensional vectors $(x_1,x_2,x_3)$. The first principal component explains about 99% variance, indicating the dimension of the original dataset is about one. I then drop one dimension (say $x_3$) from the dataset and do PCA (on reduced data $(x_1,x_2)$). This time the first principal component explains only 80% variance.

$\endgroup$
  • $\begingroup$ What exactly do you drop from the data set? $\endgroup$ – Dave Sep 21 at 19:51
  • $\begingroup$ I have added more details $\endgroup$ – J. Lin Sep 21 at 19:56
  • 2
    $\begingroup$ What were the eigenvalues and eigenvectors? If $x_3$ dominated the first PC, then of course you get a way different result when you drop it! $\endgroup$ – Dave Sep 21 at 20:05
  • $\begingroup$ Could you also check the correlation between the variables? It could be possible that if two variable are correlated strongly, they over emphasize the significance of first principle component. But you really need to check for this, other it could be you're dropping important information and hence PC1 is much smaller. $\endgroup$ – user23564 Oct 6 at 17:08
1
$\begingroup$

In PCA, you find new axes that explains the most variance. Your old axes were $x_1,x_2,x_3$; and your new axes (i.e. eigenvectors of covariance matrix) are linear combinations of these axes. For example, the principal component that can explain $99 \%$ of the variance is of the form $e_1=\alpha x_1+\beta x_2+\theta x_3$. If you drop one of the features, say $x_3$, you'll try to explain the data (i.e. $x_1,x_2$) using axes of the form $ax_1+bx_2$, which of course does not guarantee keeping $99 \%$ explainability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.