1
$\begingroup$

I have a data frame with about 500 observations and 8 variables that I'd like to run through PCA in order to try and reduce the number of variables to only those with the most variance.

From here, I want to find the [Euclidean] distance between each observation.

Here's my question: should I use every Principal Component to calculate the distances? Or should I just use (by the general rule of thumb) the Principal Components that describe, in total, about 90% of the variance (here, the first 6)?

Here's the importance of components (from R) if you're curious:

Importance of components:
                          PC1    PC2    PC3    PC4    PC5     PC6     PC7     PC8
Standard deviation     1.4652 1.1997 1.0477 0.9630 0.9103 0.87524 0.75321 0.47645
Proportion of Variance 0.2683 0.1799 0.1372 0.1159 0.1036 0.09576 0.07092 0.02838
Cumulative Proportion  0.2683 0.4482 0.5855 0.7014 0.8050 0.90071 0.97162 1.00000

Any ideas? I'd appreciate any insight.

Improve this question
$\endgroup$
  • $\begingroup$ I see you cumulative proportion sums to 1 for an over-determined system, does this mean that the values are referring to proportion of explained variance, rather than proportion of total variance? It seems unlikely that you'd strike it lucky and explained 100% of total variance with an over-determined system. Which R package are you using for PCA here? $\endgroup$ – ReneBt Mar 11 '19 at 8:59
2
$\begingroup$

If you want to get the exact Euclidean difference then you're going to have to use all principal components. Of course, the main advantage of using PCA is that you don't lose much in the approximation if you drop off the components with the low eigenvalues. So if you were to drop the components with low eigenvalues you'd still get an approximation to the exact Euclidean distance, and if the eigenvalues on those components were low it should be quite a good approximation.

Improve this answer
$\endgroup$
  • $\begingroup$ Would the distance not only be exact if the the solution is exactly-determined rather than over-determined? In an over-determined case the PC solution will give at most 8 in Evan O's case, but we have 500 observations, thus we have truncation of the PCs based on number of observations, leading to an approximation? Perhaps I am missing something. $\endgroup$ – ReneBt Mar 11 '19 at 9:05
1
$\begingroup$

Given a set of data $(x_1, x_2, \ldots, x_n) \in \mathbb{R}^p$, the principal component decomposition of rank $q\leq p$ is the projection of such points onto an affine hyperplane. The affine hyperplane is determined once one specifies an orthogonal basis $\left\{v_q\right\}$ with the property that $v_1$ contains the most variance, $v_2$ the second most and so forth. Such transformation is orthogonal if $q=p$; in such a case distances between points are preserved, by definition. If $q<p$ it needs not be so, therefore by using only the first $q$ components one obtains the distances of the projections of the points onto the affine hyperplane, not the distances of the original points.

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the answer. Unfortunately, I'm not quite sure what you're saying. If I can try and work a bit backwards, are you suggesting that I use every Principal Component in this case, or only the first 6? $\endgroup$ – Evan O. Feb 28 '19 at 5:13
  • $\begingroup$ You must use all components because only in such a case is the transformation orthogonal. $\endgroup$ – gented Feb 28 '19 at 7:58
  • $\begingroup$ Thanks for the response. In this case, is there a major advantage to finding the distance between all the PC's vs. finding the distance between all of the original variables? Apologies if that's a stupid question. $\endgroup$ – Evan O. Mar 4 '19 at 1:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.