I've got a problem I'm trying to solve at work where we have over 500 features to predict a binary outcomes (buys/ not buy). I'm being asked to throw everything into a PCA and then run a model.

There's a number of reasons I think this is a bad idea, but after reading further, I'm not too clear as to whether I'm correct. First thing is that we're going to run into multicollinearity issues where certain features may be correlated with each other causing a misinterpretation of the results. However, as I read more about PCA, because it creates orthogonal components, this may not really be an issue? I'd love to hear an answer to this.

Second, given a model with 500 features, even if we do run a PCA and end up with 3-4 components, it'll be near impossible to interpret what is actually going on. Is there a solution to this?

  • $\begingroup$ What do you want to do? Are you concerned with interpretation? Maybe yours bosses just want the model that best predicts whether someone will purchase and they don’t care about interpreting the results other than being able to say this customer will/will not purchase $\endgroup$
    – astel
    Nov 21 '19 at 21:27
  • $\begingroup$ Yeah, we want to be able to interpret the drivers of purchase behavior given the features (behaviors). However, I'm the one that will be communicating the results which is why I'm pushing back on this method -- although I don't have much say at this point. $\endgroup$
    – D500
    Nov 21 '19 at 21:33

For the question in your second paragraph, yes, PCA is one way to deal with collinearity issues and one way to reduce a large number of variables to a smaller number. There are others, including factor analysis (often giving similar results to PCA) and partial least squares (sort of like PCA but also involves the dependent variable).

Regarding the interpretability of PCA - you can't really tell, before you do the analysis, how interpretable the components will be. Sometimes, the interpretation is very clear, sometimes it is not clear at all.

As to using all 500 variables, well, one problem will be overfitting (unless your data set is quite large) and another will be colinearity, which affects the parameter estimates and their variances and therefore makes interpretation very hard (but it doesn't affect prediction). And, of course, you will have 500 parameters to interpret.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.