How to decide between PCA and logistic regression? I´ve been studying some advanced statistics, but some concepts and differences between methods are hard to grasp.
Let´s say I have a large group of individuals, each with a set of variables like age, height, IQ and so on. They can either belong to a Criminal group or NotCriminal group.
If I wanted to evaluate which characteristics are more likely to influence if someone is a criminal or not, should I use PCA or logistic regression?
 A: https://en.wikipedia.org/wiki/Principal_component_analysis

Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.

It does completely different thing to logistic regression. PCA is not a substitute for logistic regression. You could actually even use them together. PCA can be used to remove dimensions that have strong correlations before PCA transformation. PCA has actually a problem that it only looks into the dimensions, not into categories and for that reason there is linear discriminant analysis (LDA).
https://en.wikipedia.org/wiki/Linear_discriminant_analysis
LDA is not a substitution for logistic regression, but a variant to do PCA-like things for data that needs to keep categories separated, for example to make logistic regression faster/simpler.
A: The key difference between two approches


*

*PCA will NOT consider the response variable but only the variance of the independent variables. 

*Logistic Regression will consider how each independent variable impact on response variable. 


We can make an example that PCA and logistic regression will have completely different results, i.e., one method shows some feature is important, but other says opposite.
Here is how do we construct the example: Independent variable $X_1$ has very small variance (see left plot $x_1$ and $x_2$ are on different scale.), BUT it closely related to the response (from the code, you can see $y$ is assigned based on $X_1$ plus uniform noise). 
Logistic regression will say it is very important (see the summary of the model in code section), but PCA will say the opposite (see the biplot / right subfigure, the length of $X_1$ arrow is very short short.).


Code (in case you want to make the same simulation)
set.seed(0)
n_data=200
x1=rnorm(n_data,sd=0.3)
x2=rnorm(n_data,sd=1)
y=ifelse(x1+0.1*runif(n_data)>0,1,2)
par(mfrow=c(1,2),cex=1.2)
plot(x1,x2,col=y,pch=20)

summary(glm(factor(y)~x1+x2-1,family = binomial()))

pr.out=princomp(cbind(x1,x2))
biplot(pr.out,xlabs=rep("*",200))


> summary(glm(factor(y)~x1+x2-1,family = binomial()))

Call:
glm(formula = factor(y) ~ x1 + x2 - 1, family = binomial())

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-2.27753  -0.19392  -0.00118   0.05413   1.24053  

Coefficients:
   Estimate Std. Error z value Pr(>|z|)    
x1 -26.4414     4.8434  -5.459 4.78e-08 ***
x2  -0.4267     0.2975  -1.434    0.152    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 277.259  on 200  degrees of freedom
Residual deviance:  66.817  on 198  degrees of freedom
AIC: 70.817

Number of Fisher Scoring iterations: 8

A: PCA is good at streamlining numerous variables and recombining them into three Mega-variables called Principal Components.  PCA by doing so is very good at resolving multicollinearity issues within your numerous variables.  Your Principal Components will be the equivalent of indexes that are weighting your variable combinations.  One variable is typically included in all three Principal Components.  Given that, PCA is often very difficult to interpret.  It makes for pretty opaque model lacking much transparency and explanatory power.  That is especially true if you have to convey your results to anyone that is not a super-quant.  Last but not least, I don't think traditional PCA does work with a binomial variable.  Someone else, mentioned Discriminant Analysis which would work better than PCA.  But, it suffers similar issues regarding opaqueness.  But, instead of having three recombined Principal Components, it has three main Discriminants.  Those recombined variables are really difficult to interpret.   
In view of the above, Logit Regression is a far better fit for what you are trying to achieve.  This method is earmarked to handle a binomial variable exactly as you have.  And, by standardizing the regression coefficients of your variables you will be able to easily measure their relative influence on your dependent binomial variable.  
