0
$\begingroup$

I am working on analysing and visualizing a dataset having 12 features and came across PCA. I reduced the dataset to 2 principal components which together explain a variance of 18%. I was able to plot the data and see if there is a tendency to form natural clusters within the data. But I could not understand what to conclude where the first two principal components only explain 18% of the variance in the whole data.

So, what could be concluded in general when:

  • The variance suggested by the first two principal components is very high as being close 90%
  • The variance suggested by the first two principal components is very low as 18% as has happened in my case

Thinking about the above 2 questions also makes me think about the assumptions PCA makes about data? What data would not be a good fit for PCA / or when should I avoid using PCA?

Edit:

Following is the loadings table, which I computed using python's scikit as :

pca.components_.T * np.sqrt(pca.explained_variance_)

Loadings:

[[ 0.8616755  -0.15340052]
[-0.42014373  0.38166004]
[ 0.81644887 -0.21071762]
[ 0.25729266  0.37770332]
[ 0.37376327  0.205526  ]
[-0.0636729   0.71293634]
[ 0.04151499  0.79056501]
[ 0.69621113  0.32425081]
[-0.77222693  0.00931596]
[ 0.42778107 -0.05213256]
[-0.19940009 -0.53609858]]

Also, here are the first 2 principal components created from the dataset:

[[ 0.48931422 -0.23858436  0.46363166  0.14610715  0.21224658 -0.03615752
0.02357485  0.39535301 -0.43851962  0.24292133 -0.11323207]

[-0.11050274  0.27493048 -0.15179136  0.27208024  0.14805156  0.51356681
0.56948696  0.23357549  0.00671079 -0.03755392 -0.38618096]]
$\endgroup$
6
  • 1
    $\begingroup$ Think about the extreme cases. If your 12 features have no information in common, you would need 12 components to capture 100% of the information (each component just contains the information of one feature, and PCA is useless). If your 12 features have everything in common, you can fit 100% of the information into 1 component (that one component will be identical to the 12 identical features, so PCA is useless). Real-world data is usually somewhere in between, and a decision on whether PCA is worthwhile is usually based on context (what are the features) and the loadings of each feature. $\endgroup$
    – suckrates
    Commented Oct 29, 2018 at 14:32
  • $\begingroup$ @AlvaroFuentes What would you say when the variance explained by 2 primary components adds up to 18%? What should I conclude from this? $\endgroup$ Commented Oct 29, 2018 at 14:36
  • $\begingroup$ If you post the loadings table I will try to give you useful comments. $\endgroup$
    – suckrates
    Commented Oct 29, 2018 at 14:43
  • $\begingroup$ @AlvaroFuentes I posted the loadings table in my question $\endgroup$ Commented Oct 29, 2018 at 14:54
  • $\begingroup$ @AlvaroFuentes Also updated the 2 principal components $\endgroup$ Commented Oct 29, 2018 at 14:58

1 Answer 1

3
$\begingroup$

Think about the extreme cases:

  • If your 12 features have no information in common, you would need 12 components to capture 100% of the information (each component just contains the information of one feature, and PCA is useless).

  • If your 12 features have everything in common, you can fit 100% of the information into 1 component (that one component will be identical to the 12 identical features, so PCA is useless).

Real-world data is usually somewhere in between, and a decision on whether PCA is worthwhile is usually based on context (what are the features?) and the loadings of each feature.

You were wondering why the two most important components only account for 18% of the variance. The easy, general answer is that this group of 12 features don't seem to have a lot in common.

A more nuanced explanation comes from looking at the loadings...

enter image description here

...and noticing that:

  • Some features load relatively high on the first component but low on the second one (blue).
  • Some features load very low on the first component and relatively high on the second one (red).
  • Other features, like green, have relatively important proportions of their variance in both components.

So, even though the group of features is not very homogeneous, we can see that some of them are more related to each other than others.

$\endgroup$
7
  • $\begingroup$ To confirm blue and red represent heterogeneity. This is because of the high variance between the 2 principal components. On the other hand, green represents homogeneity because of the low variance between the 2 principal components. Because of the overall homogeneity (green color) being low, the variance explained by the first two principal components is very low. $\endgroup$ Commented Oct 29, 2018 at 17:56
  • $\begingroup$ Yes, that's the general idea. $\endgroup$
    – suckrates
    Commented Oct 29, 2018 at 18:16
  • $\begingroup$ Would a very low variance also mean that we would lose so many aspects of data if we try to visualize it in 2D using PCA? $\endgroup$ Commented Oct 30, 2018 at 8:40
  • 1
    $\begingroup$ Suppose features 1-6 are very weak indicators of empathy and features 7-12 are very weak indicators of conservatism. Then you observe that component 1 has high loadings from 1-6 and very low from 7-12, while component 2 behaves the opposite. Even if the two components only account for 18% of the information, you could argue that that little bit of information is exactly what you were hoping to extract from those weak indicators. $\endgroup$
    – suckrates
    Commented Oct 30, 2018 at 9:02
  • 1
    $\begingroup$ Section IV of this paper summarizes the assumptions: (1) linearity, (2) large variances have more important structure, (3) the principal components are orthogonal. Regarding bias, it doesn't make sense to talk about bias because you are not estimating anything. You are only rearranging the information of the features into new features. $\endgroup$
    – suckrates
    Commented Oct 30, 2018 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.