37
$\begingroup$

I want to reduce the dimensionality of higher order systems and capture most of the covariance on a preferably 2 dimensional or 1 dimensional field. I understand this can be done via principal component analysis, and I have used PCA in many scenarios. However, I have never used it with boolean data types, and I was wondering if it is meaningful to do PCA with this set. So for example, pretend I have qualitative or descriptive metrics, and I assign a "1" if that metric is valid for that dimension, and a "0" if it is not (binary data). So for example, pretend you are trying to compare the Seven Dwarfs in Snow White. We have:

Doc, Dopey, Bashful, Grumpy, Sneezy, Sleepy and Happy, and you want to arrange them based on qualities, and did so as is:

$$\begin{pmatrix} & Lactose\ Intolerant & A \ Honor\ Roll & Athletic & Wealthy \\ Doc & 1 & 0 & 1 & 1 \\ Dopey & 0 & 0 & 0 & 0 \\ Bashful & 1 & 0 & 1 & 1 \\ Grumpy & 1 & 1 & 1 & 1 \\ Sneezy & 0 & 1 & 1 & 0 \\ Sleepy & 1 & 0 & 0 & 0 \\ Happy & 1 & 1 & 0 & 0 \end{pmatrix}$$

So for example Bashful is lactose intolerant and not on the A honor roll. This is a purely hypothetical matrix, and my real matrix will have many more descriptive columns. My question is, would it still be appropriate to do PCA on this matrix as a means of finding the similarity between individuals?

$\endgroup$
  • 1
    $\begingroup$ This question is (almost) a duplicate of that one. PCA may be done on binary/boolean data, but doing factor analysis (including PCA "as if" it is FA) on such data is problematic. $\endgroup$ – ttnphns Jul 3 '15 at 6:50
  • 1
    $\begingroup$ PCA on binary data such as yours ("present" vs "absent") would normally be performed without centering the variables because there is no reason to suggest the origin (the reference point) other than the original 0. So, instead of covariance- or correlation-based PCA we arrive at SSCP- or cosine-based one. Such analysis is very similar, almost equivalent to Multiple Correspondence analysis (= Homogeneity analysis) which could be the choice for you. $\endgroup$ – ttnphns Jul 3 '15 at 6:58
  • $\begingroup$ a means of finding the similarity between individuals. But this task is for a Cluster analysis, not PCA. $\endgroup$ – ttnphns Jul 3 '15 at 6:59
  • $\begingroup$ Short answer: linear PCA (if it is taken as dimensionality reduction technique and not latent variable technique as factor analysis) can be used for scale (metrical) or binary data. Plain (linear) PCA should not be used, however, with ordinal data or nominal data - unless these data are turned into metrical or binary (e.g. dummy) some way. $\endgroup$ – ttnphns Jul 3 '15 at 7:11
  • $\begingroup$ @ttnphns PCA can be viewed as a way to cluster variables. Also, PCA and cluster analysis can be used in sequence $\endgroup$ – Antoine Jul 3 '15 at 9:54
12
$\begingroup$

I would like to suggest you a relatively recent technique for automatic structure extraction from categorical variable data (this includes binary). The method is called CorEx from Greg van Steeg from University of Southern California. The idea is to use the notion of Total Correlation based on the entropy measures. It is appealing due to its simplicity and no tuning of large number of hyperparameters.

The paper about hierarchical representations (the most recent, builds on the top of the previous measures). http://arxiv.org/pdf/1410.7404.pdf

$\endgroup$
13
$\begingroup$

You can also use Multiple Correspondence Analysis (MCA), which is an extension of principal component analysis when the variables to be analyzed are categorical instead of quantitative (which is the case here with your binary variables). See for instance Husson et al. (2010), or Abdi and Valentin (2007). An excellent R package to perform MCA (and hierarchical clustering on PCs) is FactoMineR.

$\endgroup$
  • 1
    $\begingroup$ An interesting answer, I'd rather agree with it. It will be great asset to it if you explain in more detail the phenomenon of the inflation of the feature space and why it would occur in PCA and not in MCA. $\endgroup$ – ttnphns Jul 3 '15 at 10:19
  • $\begingroup$ I misunderstood the inflation of the feature space phenomenon. It seems to be coming into play when going from CA to MCA, but is not an inherent issue of PCA. I am going to remove my answer when you have read this comment. Thanks for making me realize that. $\endgroup$ – Antoine Jul 3 '15 at 10:47
  • $\begingroup$ I don't think that you have to remove the answer. MCA is one of right choices, for me, and your answer is all right. $\endgroup$ – ttnphns Jul 3 '15 at 10:54
  • $\begingroup$ I added back the answer because I think MCA might be helpful here, but I removed the discussion about the inflation of the feature space since it did not seem to be relevant $\endgroup$ – Antoine Jul 6 '15 at 8:56
  • $\begingroup$ How would the result of MCA on binary data differ from the result of a PCoA with a distance measure appropriate for binary data like Jaccard or simple matching? $\endgroup$ – emudrak Aug 16 '17 at 20:48
9
$\begingroup$

If you think of PCA as an exploratory technique to give you a way to visualise the relationships between variables (and in my opinion this is the only way to think about it) then yes, there is no reason why you can't put in binary variables. For example, here is a biplot of your data

enter image description here

It seems reasonably useful. For example, you can see that Doc and Bashful are very similar; that HR is rather unlike the three other variables; Sleepy and Sneezy are very dissimilar, etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.