Apologies if this question has been asked before, but I could not find very relevant topics.

I am working with proteomic data (40 proteins, 800 instances) where the outcome variable is binary (presence or absence of a disease). The question is very simple: build the best classification model and find the subset of the most contributing features (proteins). That's fine. I used several standard feature selection approaches (L1/2, elastic net and some others).

However, I was told that instead of applying feature selection methods, I should have used the univariate analysis (significance test of each protein and the outcome variable) and based on the p-values (adjusted to multiple comparisons), I should select 'the best subset'.

When I performed this univariate analysis and used the subselected proteins based on small p-value to build the classification model, I got worse results (AUC, F1, etc), when I used regularisation schemes and feature selection.

QUESTION: Why subset of features selected through the univariate significance test is not as good as the one selected by LASSO or similar? What is the statistical picture behind it?

Any links to relevant papers will be highly appreciated.

  • 3
    $\begingroup$ Honestly, who gave you the advice, to use a number of univariate analyses for multivariate data? That is an approach that cannot take dependencies between the data, is sensitive to pseudo-correlations, probably uses arbitrary cut-off values etc. $\endgroup$ – Bernhard Feb 12 '18 at 13:19
  • $\begingroup$ @Bernhard, I can't agree more, but I am new to biostats (coming from maths. and ML), and the person was very confident about that. Also I saw some papers where this approach was used for feature selection in multivariate data. Any links/ideas to dig deeper on the topic? $\endgroup$ – Arnold Klein Feb 12 '18 at 13:21
  • 3
    $\begingroup$ You have not started the thinking process off very well. Question why feature selection is an important thing to do, vs. getting the best predictions. And question classification, probably using prediction instead. More at fharrell.com/post/classification . $\endgroup$ – Frank Harrell Feb 12 '18 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.