5
$\begingroup$

I´ve been studying some advanced statistics, but some concepts and differences between methods are hard to grasp.

Let´s say I have a large group of individuals, each with a set of variables like age, height, IQ and so on. They can either belong to a Criminal group or NotCriminal group.

If I wanted to evaluate which characteristics are more likely to influence if someone is a criminal or not, should I use PCA or logistic regression?

Improve this question
$\endgroup$
4
  • 11
    $\begingroup$ Since "likely" and "influence" are concepts entirely foreign to what PCA does, could you elaborate a little on how you conceive of applying PCA to this question? $\endgroup$
    – whuber ♦
    Commented Nov 7, 2016 at 15:01
  • $\begingroup$ If i had 20 variables, wouldn´t using PCA help summarizing the info and check which PCs influence variance the most? As I understand it, in the end wouldn´t I get a graph with two clusters Criminal and NonCriminal? $\endgroup$
    – DrGenius
    Commented Nov 7, 2016 at 15:06
  • $\begingroup$ Both of these have a notion of linearity and additivity to them and I guess, the "true" solution will be non-linear. Have you considered tree methods or randomForests, other machine learning methods? $\endgroup$
    – Bernhard
    Commented Nov 7, 2016 at 15:18
  • 2
    $\begingroup$ You would not necessarily get "get a graph with two clusters" even if the variables were related to criminality. That said, PCA can possibly be combined with logistic regression in a helpful way. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 7, 2016 at 15:20

3 Answers 3

Reset to default
23
$\begingroup$

The key difference between two approches

  • PCA will NOT consider the response variable but only the variance of the independent variables.
  • Logistic Regression will consider how each independent variable impact on response variable.

We can make an example that PCA and logistic regression will have completely different results, i.e., one method shows some feature is important, but other says opposite.

Here is how do we construct the example: Independent variable $X_1$ has very small variance (see left plot $x_1$ and $x_2$ are on different scale.), BUT it closely related to the response (from the code, you can see $y$ is assigned based on $X_1$ plus uniform noise).

Logistic regression will say it is very important (see the summary of the model in code section), but PCA will say the opposite (see the biplot / right subfigure, the length of $X_1$ arrow is very short short.).

enter image description here

Code (in case you want to make the same simulation)

set.seed(0)
n_data=200
x1=rnorm(n_data,sd=0.3)
x2=rnorm(n_data,sd=1)
y=ifelse(x1+0.1*runif(n_data)>0,1,2)
par(mfrow=c(1,2),cex=1.2)
plot(x1,x2,col=y,pch=20)

summary(glm(factor(y)~x1+x2-1,family = binomial()))

pr.out=princomp(cbind(x1,x2))
biplot(pr.out,xlabs=rep("*",200))


> summary(glm(factor(y)~x1+x2-1,family = binomial()))

Call:
glm(formula = factor(y) ~ x1 + x2 - 1, family = binomial())

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-2.27753  -0.19392  -0.00118   0.05413   1.24053  

Coefficients:
   Estimate Std. Error z value Pr(>|z|)    
x1 -26.4414     4.8434  -5.459 4.78e-08 ***
x2  -0.4267     0.2975  -1.434    0.152    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 277.259  on 200  degrees of freedom
Residual deviance:  66.817  on 198  degrees of freedom
AIC: 70.817

Number of Fisher Scoring iterations: 8
Improve this answer
$\endgroup$
3
  • $\begingroup$ That last paragraph really added to my understanding of both concepts, I´m slowly getting it. $\endgroup$
    – DrGenius
    Commented Nov 7, 2016 at 16:16
  • 3
    $\begingroup$ PCA is one of those things, like Naive Bayes, that really should not work in the real world, and yet it seems to. As you say, PCA does not take into account the target, so the fact that PCA seems to help us classify/cluster things is a serendipitous thing. It's trivial to make a counter-example where the first principle component is exactly wrong. $\endgroup$
    – Wayne
    Commented Nov 7, 2016 at 22:52
  • $\begingroup$ While the general concept of your answer is correct (PCA being an unsupervised method does not guarantee selection of latent vectors in any way associated with the response variable), your example is misleading. This is largely because your input to PCA is not standardized. The reason why x2 is much more "important" that x1 is simply because of its larger variance. If the input had been centered and standardized, x1 and x2 would have roughly similar (yet orthogonal) vectors in the right panel. $\endgroup$
    – afrendeiro
    Commented Mar 13, 2019 at 13:33
4
$\begingroup$

https://en.wikipedia.org/wiki/Principal_component_analysis

Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.

It does completely different thing to logistic regression. PCA is not a substitute for logistic regression. You could actually even use them together. PCA can be used to remove dimensions that have strong correlations before PCA transformation. PCA has actually a problem that it only looks into the dimensions, not into categories and for that reason there is linear discriminant analysis (LDA). https://en.wikipedia.org/wiki/Linear_discriminant_analysis

LDA is not a substitution for logistic regression, but a variant to do PCA-like things for data that needs to keep categories separated, for example to make logistic regression faster/simpler.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I think I understand what you mean. In this case, then, PCA should be used to process the data before subjecting it to logistic regression, since, height and IQ may be related, for example? $\endgroup$
    – DrGenius
    Commented Nov 7, 2016 at 15:34
  • $\begingroup$ The function of the two are completely different. PCA is bad for the reason I said. LDA is invented for dimension reduction, especially because PCA can mess data in your particular application. $\endgroup$
    – user3644640
    Commented Nov 7, 2016 at 15:39
2
$\begingroup$

PCA is good at streamlining numerous variables and recombining them into three Mega-variables called Principal Components. PCA by doing so is very good at resolving multicollinearity issues within your numerous variables. Your Principal Components will be the equivalent of indexes that are weighting your variable combinations. One variable is typically included in all three Principal Components. Given that, PCA is often very difficult to interpret. It makes for pretty opaque model lacking much transparency and explanatory power. That is especially true if you have to convey your results to anyone that is not a super-quant. Last but not least, I don't think traditional PCA does work with a binomial variable. Someone else, mentioned Discriminant Analysis which would work better than PCA. But, it suffers similar issues regarding opaqueness. But, instead of having three recombined Principal Components, it has three main Discriminants. Those recombined variables are really difficult to interpret.

In view of the above, Logit Regression is a far better fit for what you are trying to achieve. This method is earmarked to handle a binomial variable exactly as you have. And, by standardizing the regression coefficients of your variables you will be able to easily measure their relative influence on your dependent binomial variable.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.