How can I tell that there is no pattern in the PCA results?

Question

I have a 1000+ samples dataset of 19 variables. My objective is to predict a binary variable based on the other 18 variables (binary and continuous). I'm quite confident that 6 of the predicting variables are associated with the binary response, however, I would like to further analyse the dataset and look for other associations or structures that I might be missing. In order to do this, I decided to use PCA and clustering.

When running the PCA on the normalized data, turns out that 11 components need to be kept in order to retain 85% of the variance. By plotting the pairplots I get this:

I'm not sure on what's next... I see no significant pattern in the pca and I am wondering what this means and if it could have been caused by the fact that some of the variables are binary. By running a clustering algorithm with 6 clusters I get the following result which is not exactly an improvement although some blobs seem to stand out (the yellow ones).

As you probably can tell, I'm not an expert on PCA, but saw some tutorials and how it can be powerful to get a glimpse of structures in high dimensional space. With the famous MNIST digits (or the IRIS) dataset it works great. My question is: what should I do now to make more sense out of the PCA? Clustering does not seem to pick up anything useful, how can I can I tell that there's no pattern in the PCA or what should I try next to find patterns in the PCA data?

Answer 1

You explained variance plot tells me that PCA is pointless here. 11/18 is 61%, so you need 61% of your variables to explain 85% of variance. That's not the case for PCA, in my opinion. I use PCA when 3-5 factors of 18 explain 95% or so of the variance.

UPDATE: Look at the plot of cumulative percent of variance explained by the number of PCs. This is from interest rate term structure modeling field. You see how 3 components explain more than 99% of total variance. This may look like a made up example for PCA advertising :) However, this is a real thing. Interest rate tenors are that much correlated, that's why PCA is very natural in this application. Instead of dealing with a couple of dozens of tenors, you deal with just 3 components.

Answer 2

If you have $N>1000$ samples and only $p=19$ predictors it would be pretty reasonable to just use all predictors in a model. In that case a PCA step may well be unnecessary.

If you are confident that only a subset of the variables are really explanatory, using a sparse regression model, e.g. Elastic Net, could help you to establish this.

Also, interpretation of PCA results using mixed type inputs (binary vs real, different scales etc, see CV question here) is not so straightforward and you may wish to avoid it unless there is a clear reason to do so.

Answer 3

I'm going to interpret your question as succinctly as I can. Let me know if it changes your meaning.

I'm quite confident that 6 of the predicting variables are associated with the binary response [but] I see no significant pattern in the pca

I don't see any "significant pattern" either, other than the consistency in your pairplots. They're all just roughly circular blobs. I'm curious what you expected to see. Clearly separate point clusters some of the pairplots? A few plots very close to linear?

Your PCA results - the bloblike pairplots and only 85% of variance captured in the top 11 principal components - don't preclude your hunch about 6 variables being sufficient for binary response prediction.

Imagine these situations:

1. Say your PCA results show that 99% of variance is captured by 6 principal components.

   That might seem to support your hunch about 6 predictor variables - maybe you could define a plane or some other surface in that 6 dimensional space which classifies the points very well, and you could use that surface as a binary predictor. Which brings me to number 2...

2. Say your top 6 principal components have pairplots that look like this

   

   But let's color code an arbitrary binary response

   

   Even though you managed to capture nearly all (99%) of the variance in 6 variables, you still aren't guaranteed to have spatial separation to predict your binary response.

You might actually need several numerical thresholds (which could be plotted as surfaces in that 6 dimensional space), and a point's membership to your binary classification might depend on a complex conditional expression made of that point's relationship to each of those thresholds. But that's just an example of how a binary class could be predicted. There are a ton of data structures and methods for representing, training, and predicting. This is a teaser. To quote,

Often the hardest part of solving a machine learning problem can be finding the right estimator for the job.