What should I look for in my PCA?

I'm doing supervised learning with (unfortunately only) 2000 examples, evenly split into 1000 yes and 1000 no. Each vector is a 1000 dimensional boolean vector. I have not run mean normalization or other normalizations on the data set.

I ran PCA and got the following. For the first three dimensions, I got:

$$\text{explained variance ratio} = \left[ \texttt{0.0193} \; \texttt{0.00933} \; \texttt{0.00817} \right]$$

I have a video in 3D of the plot here, but for simplicity, here the data set is post PCA in 2 dimensions (red means yes, blue means no).

Question: Does this dataset seem to be non-learnable? How can I tell?

PCA in 2D (full version)

  • 3
    $\begingroup$ Lot's of people do it but I'm not sure PCA is the best method to apply to binary data. You are also throwing away much information with only 3 PCs, try other techniques if possible and also regularized learners. Perhaps your data is also sparse, and you could try methods tailored to that kind of data. $\endgroup$ – Firebug May 29 '16 at 17:03

No, I'm inclined to be optimistic. The top half of the plot has more red than blue dots, and the bottom half has more blue than red dots, so you should be able to predict the dependent variable more accurately than the base rate of 50%.

  • $\begingroup$ +1. Also, looking at the figure, LDA could be substituted (or combined) with PCA. $\endgroup$ – jeff May 29 '16 at 19:30

I agree with Kodiologist. The Red vs. Blue areas have enough of an area that does not perfectly overlap to hopefully be differentiable from one another. Given the binomial nature of the data, I would use quantitative methods better catered to such data shape such as Logit Regression or Probit Regression. I think those methods are much superior to PCA given the data structure. I am not even sure PCA is appropriate at all for such data structure.

  • 2
    $\begingroup$ (1) PCA with binary data is not particularly problematic. (2) Just how do you propose to apply regression to a dataset with a thousand explanatory variables and just two thousand observations? On the face of it there's little hope of success: some kind of dimension reduction technique, regularization, or alternative procedure (such as SVM or Random Forest) would seem more attractive. $\endgroup$ – whuber May 29 '16 at 21:32
  • $\begingroup$ whuber, I have read the interesting answer you are referring to suggesting that PCA could work with binary data. However, wouldn't PCA be subject to the same flaws as linear regression that you can actually get values < 0 and > 1. $\endgroup$ – Sympa May 30 '16 at 0:30
  • 1
    $\begingroup$ I'm not sure what "values" you are referring to with PCA. It doesn't predict anything--at most, it approximately reconstructs the data. For the purposes of an approximation there is no problem with values outside the original range of data values. $\endgroup$ – whuber May 30 '16 at 17:46
  • $\begingroup$ PCA just aggregates tens of different variables into three aggregate ones (the Principal Components) when modeling a dependent variable. In essence, it streamlines your multiple regression that becomes completely unfeasible with too many variables that inevitably have some multicollinearity issues. But, the PCA mechanics are not that different from regression. And, you can't use regular regression on a binomial variable (0, 1) because you get values < 0 and > 1. The binomial variable can be interpreted as a probability. And, probabilities < 0 and >1 just don't make sense. $\endgroup$ – Sympa May 30 '16 at 22:19
  • 1
    $\begingroup$ Unfortunately, little of that comment makes sense, either. There is no obligation to interpret binomial variables as probabilities, nor is there any restriction of PCA to three "aggregate" variables, nor is it a form of "multiple regression." $\endgroup$ – whuber May 31 '16 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.