I have a 1000+ samples dataset of 19 variables. My objective is to predict a binary variable based on the other 18 variables (binary and continuous). I'm quite confident that 6 of the predicting variables are associated with the binary response, however, I would like to further analyse the dataset and look for other associations or structures that I might be missing. In order to do this, I decided to use PCA and clustering.

When running the PCA on the normalized data, turns out that 11 components need to be kept in order to retain 85% of the variance. enter image description here By plotting the pairplots I get this: enter image description here

I'm not sure on what's next... I see no significant pattern in the pca and I am wondering what this means and if it could have been caused by the fact that some of the variables are binary. By running a clustering algorithm with 6 clusters I get the following result which is not exactly an improvement although some blobs seem to stand out (the yellow ones). enter image description here

As you probably can tell, I'm not an expert on PCA, but saw some tutorials and how it can be powerful to get a glimpse of structures in high dimensional space. With the famous MNIST digits (or the IRIS) dataset it works great. My question is: what should I do now to make more sense out of the PCA? Clustering does not seem to pick up anything useful, how can I can I tell that there's no pattern in the PCA or what should I try next to find patterns in the PCA data?

  • $\begingroup$ Why are you doing PCA in order to find predictors? Why not use some other method? e.g you could include them all in a logistic reg, you could use LASSO, you could build a tree model, there's bagging, boosting etc. $\endgroup$ – Peter Flom Sep 16 '15 at 13:32
  • $\begingroup$ What do specifically you mean by "pattern" that PCA is good to reveal at? $\endgroup$ – ttnphns Sep 16 '15 at 13:35
  • $\begingroup$ @ttnphns what I'm trying to do is to find some subgroup of observations that may have something in common to better explain the outcome of the binary response I'm trying to predict (this has been partly inspired by everydayanalytics.ca/2014/06/…). Also using pca and clustering on the iris dataset it is useful to isolate the species (scikit-learn.org/stable/auto_examples/decomposition/…) although that is super-easy since we already know the number of clusters. $\endgroup$ – mickkk Sep 16 '15 at 13:45
  • $\begingroup$ @PeterFlom I've already run logistic regression and a random forest model and they're performing decently, however I would like to further investigate the data. $\endgroup$ – mickkk Sep 16 '15 at 13:47

You explained variance plot tells me that PCA is pointless here. 11/18 is 61%, so you need 61% of your variables to explain 85% of variance. That's not the case for PCA, in my opinion. I use PCA when 3-5 factors of 18 explain 95% or so of the variance.

UPDATE: Look at the plot of cumulative percent of variance explained by the number of PCs. This is from interest rate term structure modeling field. You see how 3 components explain more than 99% of total variance. This may look like a made up example for PCA advertising :) However, this is a real thing. Interest rate tenors are that much correlated, that's why PCA is very natural in this application. Instead of dealing with a couple of dozens of tenors, you deal with just 3 components.

enter image description here

| cite | improve this answer | |
  • $\begingroup$ That's what I suspected in the first place. I did not point it out directly because I do not know that much about PCA to make such a bold statement. Is it safe to say that when more than x% of the components are needed then PCA is not much of an help? I mean, in the examples of application I saw, usually few components explain the most variance. $\endgroup$ – mickkk Sep 16 '15 at 15:12
  • $\begingroup$ @mickkk, there's no firm rule. To me the indication is convexity the variance explained graph. If you draw it as a cumulative percentage of the total variance explained by number of PCs, then you want to see a very concave graph. Your would have been close to linear: each component seems to carry roughly the same information about the data, in this case why use PCA at all instead of the original data? $\endgroup$ – Aksakal Sep 16 '15 at 16:57
  • $\begingroup$ The edit with the new example was very helpful. $\endgroup$ – mickkk Sep 16 '15 at 18:34

If you have $N>1000$ samples and only $p=19$ predictors it would be pretty reasonable to just use all predictors in a model. In that case a PCA step may well be unnecessary.

If you are confident that only a subset of the variables are really explanatory, using a sparse regression model, e.g. Elastic Net, could help you to establish this.

Also, interpretation of PCA results using mixed type inputs (binary vs real, different scales etc, see CV question here) is not so straightforward and you may wish to avoid it unless there is a clear reason to do so.

| cite | improve this answer | |

I'm going to interpret your question as succinctly as I can. Let me know if it changes your meaning.

I'm quite confident that 6 of the predicting variables are associated with the binary response [but] I see no significant pattern in the pca

I don't see any "significant pattern" either, other than the consistency in your pairplots. They're all just roughly circular blobs. I'm curious what you expected to see. Clearly separate point clusters some of the pairplots? A few plots very close to linear?

Your PCA results - the bloblike pairplots and only 85% of variance captured in the top 11 principal components - don't preclude your hunch about 6 variables being sufficient for binary response prediction.

Imagine these situations:

  1. Say your PCA results show that 99% of variance is captured by 6 principal components.

    That might seem to support your hunch about 6 predictor variables - maybe you could define a plane or some other surface in that 6 dimensional space which classifies the points very well, and you could use that surface as a binary predictor. Which brings me to number 2...

  2. Say your top 6 principal components have pairplots that look like this

    "Pattern" in pairplots.

    But let's color code an arbitrary binary response

    "Pattern" is useless.

    Even though you managed to capture nearly all (99%) of the variance in 6 variables, you still aren't guaranteed to have spatial separation to predict your binary response.

You might actually need several numerical thresholds (which could be plotted as surfaces in that 6 dimensional space), and a point's membership to your binary classification might depend on a complex conditional expression made of that point's relationship to each of those thresholds. But that's just an example of how a binary class could be predicted. There are a ton of data structures and methods for representing, training, and predicting. This is a teaser. To quote,

Often the hardest part of solving a machine learning problem can be finding the right estimator for the job.

| cite | improve this answer | |
  • 1
    $\begingroup$ Smiley face is actually good, because it is uncorrelated! I liked it. $\endgroup$ – amoeba Sep 16 '15 at 19:18
  • $\begingroup$ @amoeba, can you have smiley face from uncorrelated PCs? $\endgroup$ – Aksakal Sep 16 '15 at 19:43
  • $\begingroup$ @Aksakal, yes, the smiley scatter plot seems to me to exhibit zero correlation. Kdbanman, I appreciate the update, +1. $\endgroup$ – amoeba Sep 16 '15 at 19:55
  • $\begingroup$ @amoeba, ok, you mean linear correlation. $\endgroup$ – Aksakal Sep 16 '15 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.