Suppose we have a two groups of people, A and B and we have measured 50 features, e.g. blood markers in these people and we would like to know which features are different between the two groups and statistically significant. Usuaully I would apply a statistics test like Mann-Whitney-U on the distribution of each feature between the two groups, and then use a procedure to correct for multiple hypothesis testing on the p-values obtained from this.

Another way could be to make a binary logistic classifier, e.g. glm in R to classify if a person belongs to group A or not. This model would then report p-values for the coefficient for each feature. First question: should these p-values be corrected for multiple hypothesis testing? I would think so. Second question: Instead of supplying all features of the data to the logistic model, what if I fit a model for each feature, obtain the p-value for that feature, and then correct these p-values for multiple hypothesis testing. Will this yield the same result - p-values - as supplying all features to the model? I guess not - and then why is this last approach not correct? It seems like this corresponds more to the first 'simple' statistical test approach? Would I be violating some assumption of the model when doing this?


1 Answer 1


If you want to discover associations between many variables and a binary outcome, you can fit tens or hundreds or thousands of bivariate logistic regression models, but for any given model's p-value to mean anything, you have to control the familywise error rate. It wouldn't make sense in this case to include all the variables in a single model. This is the only setting you described in which a p-value is being used for its true purpose: as a measure of significance with a well defined null-hypothesis.

Alternately, if the goal were to predict the group membership, the goal is now prediction rather than inference. You could throw all the variables into a single model. In that case, we wouldn't care what the p-value was for any variable because the p-value is an imprecise measure of predictiveness. You could enforce parsimony by performing backward (or forward) stepwise model selection, thereby deleting non-significant variables. This model will have lower internal validity, but hopefully a higher external validity. But again, the p-value is only incidentally being used to encourage generalizability. You may just as well do the same thing using penalized regression.

In general, feature selection is not a shortcut to statistical inference. A variable may be predictive but not statistically significant, or vice versa. A p-value becomes increasingly hard to conceptualize when the space of possible hypotheses grows at $2^K$ where $K$ is the number of features considered in an analysis.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.