The key difference between two approches
- PCA will NOT consider the response variable but only the variance of the independent variables.
- Logistic Regression will consider how each independent variable impact on response variable.
We can make an example that PCA and logistic regression will have completely different results, i.e., one method shows some feature is important, but other says opposite.
Here is how do we construct the example: Independent variable $X_1$ has very small variance (see left plot $x_1$ and $x_2$ are on different scale.), BUT it closely related to the response (from the code, you can see $y$ is assigned based on $X_1$ plus uniform noise).
Logistic regression will say it is very important (see the summary of the model in code section), but PCA will say the opposite (see the biplot / right subfigure, the length of $X_1$ arrow is very short short.).
Code (in case you want to make the same simulation)
summary(glm(factor(y)~x1+x2-1,family = binomial()))
> summary(glm(factor(y)~x1+x2-1,family = binomial()))
glm(formula = factor(y) ~ x1 + x2 - 1, family = binomial())
Min 1Q Median 3Q Max
-2.27753 -0.19392 -0.00118 0.05413 1.24053
Estimate Std. Error z value Pr(>|z|)
x1 -26.4414 4.8434 -5.459 4.78e-08 ***
x2 -0.4267 0.2975 -1.434 0.152
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 277.259 on 200 degrees of freedom
Residual deviance: 66.817 on 198 degrees of freedom
Number of Fisher Scoring iterations: 8