0
$\begingroup$

I am working on analysing and visualizing a dataset having 12 features and came across PCA. I reduced the dataset to 2 principal components which together explain a variance of 18%. I was able to plot the data and see if there is a tendency to form natural clusters within the data. But I could not understand what to conclude where the first two principal components only explain 18% of the variance in the whole data.

So, what could be concluded in general when:

  • The variance suggested by the first two principal components is very high as being close 90%
  • The variance suggested by the first two principal components is very low as 18% as has happened in my case

Thinking about the above 2 questions also makes me think about the assumptions PCA makes about data? What data would not be a good fit for PCA / or when should I avoid using PCA?

Edit:

Following is the loadings table, which I computed using python's scikit as :

pca.components_.T * np.sqrt(pca.explained_variance_)

Loadings:

[[ 0.8616755  -0.15340052]
[-0.42014373  0.38166004]
[ 0.81644887 -0.21071762]
[ 0.25729266  0.37770332]
[ 0.37376327  0.205526  ]
[-0.0636729   0.71293634]
[ 0.04151499  0.79056501]
[ 0.69621113  0.32425081]
[-0.77222693  0.00931596]
[ 0.42778107 -0.05213256]
[-0.19940009 -0.53609858]]

Also, here are the first 2 principal components created from the dataset:

[[ 0.48931422 -0.23858436  0.46363166  0.14610715  0.21224658 -0.03615752
0.02357485  0.39535301 -0.43851962  0.24292133 -0.11323207]

[-0.11050274  0.27493048 -0.15179136  0.27208024  0.14805156  0.51356681
0.56948696  0.23357549  0.00671079 -0.03755392 -0.38618096]]
$\endgroup$
  • 1
    $\begingroup$ Think about the extreme cases. If your 12 features have no information in common, you would need 12 components to capture 100% of the information (each component just contains the information of one feature, and PCA is useless). If your 12 features have everything in common, you can fit 100% of the information into 1 component (that one component will be identical to the 12 identical features, so PCA is useless). Real-world data is usually somewhere in between, and a decision on whether PCA is worthwhile is usually based on context (what are the features) and the loadings of each feature. $\endgroup$ – Alvaro Fuentes Oct 29 '18 at 14:32
  • $\begingroup$ @AlvaroFuentes What would you say when the variance explained by 2 primary components adds up to 18%? What should I conclude from this? $\endgroup$ – Suhail Gupta Oct 29 '18 at 14:36
  • $\begingroup$ If you post the loadings table I will try to give you useful comments. $\endgroup$ – Alvaro Fuentes Oct 29 '18 at 14:43
  • $\begingroup$ @AlvaroFuentes I posted the loadings table in my question $\endgroup$ – Suhail Gupta Oct 29 '18 at 14:54
  • $\begingroup$ @AlvaroFuentes Also updated the 2 principal components $\endgroup$ – Suhail Gupta Oct 29 '18 at 14:58
2
$\begingroup$

Think about the extreme cases:

  • If your 12 features have no information in common, you would need 12 components to capture 100% of the information (each component just contains the information of one feature, and PCA is useless).

  • If your 12 features have everything in common, you can fit 100% of the information into 1 component (that one component will be identical to the 12 identical features, so PCA is useless).

Real-world data is usually somewhere in between, and a decision on whether PCA is worthwhile is usually based on context (what are the features?) and the loadings of each feature.

You were wondering why the two most important components only account for 18% of the variance. The easy, general answer is that this group of 12 features don't seem to have a lot in common.

A more nuanced explanation comes from looking at the loadings...

enter image description here

...and noticing that:

  • Some features load relatively high on the first component but low on the second one (blue).
  • Some features load very low on the first component and relatively high on the second one (red).
  • Other features, like green, have relatively important proportions of their variance in both components.

So, even though the group of features is not very homogeneous, we can see that some of them are more related to each other than others.

$\endgroup$
  • $\begingroup$ To confirm blue and red represent heterogeneity. This is because of the high variance between the 2 principal components. On the other hand, green represents homogeneity because of the low variance between the 2 principal components. Because of the overall homogeneity (green color) being low, the variance explained by the first two principal components is very low. $\endgroup$ – Suhail Gupta Oct 29 '18 at 17:56
  • $\begingroup$ Yes, that's the general idea. $\endgroup$ – Alvaro Fuentes Oct 29 '18 at 18:16
  • $\begingroup$ Would a very low variance also mean that we would lose so many aspects of data if we try to visualize it in 2D using PCA? $\endgroup$ – Suhail Gupta Oct 30 '18 at 8:40
  • 1
    $\begingroup$ Suppose features 1-6 are very weak indicators of empathy and features 7-12 are very weak indicators of conservatism. Then you observe that component 1 has high loadings from 1-6 and very low from 7-12, while component 2 behaves the opposite. Even if the two components only account for 18% of the information, you could argue that that little bit of information is exactly what you were hoping to extract from those weak indicators. $\endgroup$ – Alvaro Fuentes Oct 30 '18 at 9:02
  • 1
    $\begingroup$ Section IV of this paper summarizes the assumptions: (1) linearity, (2) large variances have more important structure, (3) the principal components are orthogonal. Regarding bias, it doesn't make sense to talk about bias because you are not estimating anything. You are only rearranging the information of the features into new features. $\endgroup$ – Alvaro Fuentes Oct 30 '18 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.