0
$\begingroup$

I'm doing some PCA on scaled features but where I also have some binary variables.

When I include the binary features they seem to really impact the PCA and I'm concerned that they will also disproportionately impact the cluster analysis I'm planning to do with kmeans too.

library(tidyverse)
library(factoextra)
df <- diamonds %>% select(depth:z) %>% 
  lapply(function(x) {(x - min(x)) / (max(x) - min(x))}) %>% as.data.frame()
df$cut = diamonds$cut
df_small <- df %>% sample_n(1000)
clust_fact <- df_small$cut %>% factor()
df_small <- df_small %>% select(-cut)
df.pc <- princomp(df_small)

pc1_cont <- fviz_contrib(df.pc, choice = "var", axes = 1)

When you then enter pc1_cont into the console you see: enter image description here

This is the contribution of each feature to PC1. You can see that price shows the most impact to PC1.

Now, if I add a binary feature, look what happens:

df <- diamonds %>% select(depth:z) %>% 
  mutate(carat_binary = if_else(diamonds$carat >= 0.8, 1, 0)) %>% # add binary feature
  lapply(function(x) {(x - min(x)) / (max(x) - min(x))}) %>% as.data.frame() # scale between 0 and 1
df$cut = diamonds$cut
df_small <- df %>% sample_n(1000)
clust_fact <- df_small$cut %>% factor()
df_small <- df_small %>% select(-cut)
df.pc <- princomp(df_small)


pc1_cont <- fviz_contrib(df.pc, choice = "var", axes = 1)

enter image description here

The new binary feature is shown to have the largest impact. This echos what I've found on my own actual data, each time I include a binary it shows as having the most impact.

I found some posts on here for handling this but struggled to understand the guidance. I was pointed to academic papers.

Is there a conventional approach to 'taking the edge off' binary features in PCA and clustering? In my mind it makes sense that they distort things since they will always be at the extreme of the sale of the entire data set, 0 or 1.

I was thinking of doing something crude like just transforming the true case as being the mean of across all scaled numeric variables in my data frame and then the false case as perhaps the 1st quartile value. But I'm thinking of arbitrary solutions here.

Is there a straight forwards approach in r for dealing with this?

$\endgroup$

closed as off-topic by Sycorax, Michael Chernick, Peter Flom Aug 8 at 11:30

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ What problem are you trying to solve? You haven't described why it's a Bad Thing for the binary features to strongly influence PCA. $\endgroup$ – Sycorax Aug 7 at 21:16
  • 2
    $\begingroup$ I'm trying to prevent my analysis from being too strongly influenced by the addition of 2 binary features out of a total of ~50 features. I'm clustering and I don't want the clusters to be too heavily determined by these binary vars $\endgroup$ – Doug Fir Aug 7 at 21:39
  • 1
    $\begingroup$ Why are you trying to do this? As far as I can tell, doing so will make your cluster analysis worse. And what is "too heavily"? $\endgroup$ – Peter Flom Aug 8 at 11:29
  • 1
    $\begingroup$ There is the hidden underlying assumption in PCA that all features are continuous, linear (price probably is too skewed!) and of equal importance and scale.p (price and x likely just cannot be compared). $\endgroup$ – Anony-Mousse Aug 10 at 7:15
  • 1
    $\begingroup$ Standardization is a different way of scaling.not to [0:1], but based in the standard deviation. Note that I'm not saying it will be better, just different. The problem is that on your data, your problem is ill-defined already. So there is no "right" way. $\endgroup$ – Anony-Mousse Aug 10 at 18:19
1
$\begingroup$

I quote your comment “Is there a conventional approach to 'taking the edge off' binary features in PCA and clustering? In my mind it makes sense that they distort things since they will always be at the extreme of the sale of the entire data set, 0 or 1.” .... Exactly that is why in one of my answer Principle Component Analysis on categorical predictors I suggested that if they are just a few, leave them aside when computing PCA and transforming the other features especially if they are not very correlated between each other and do not need simplification. Look also here for a good discussion on the topic Doing principal component analysis or factor analysis on binary data

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.