Apologies if this question has been asked before, but I could not find very relevant topics.
I am working with proteomic data (40 proteins, 800 instances) where the outcome variable is binary (presence or absence of a disease). The question is very simple: build the best classification model and find the subset of the most contributing features (proteins). That's fine. I used several standard feature selection approaches (L1/2, elastic net and some others).
However, I was told that instead of applying feature selection methods, I should have used the univariate analysis (significance test of each protein and the outcome variable) and based on the p-values (adjusted to multiple comparisons), I should select 'the best subset'.
When I performed this univariate analysis and used the subselected proteins based on small p-value to build the classification model, I got worse results (AUC, F1, etc), when I used regularisation schemes and feature selection.
QUESTION: Why subset of features selected through the univariate significance test is not as good as the one selected by LASSO or similar? What is the statistical picture behind it?
Any links to relevant papers will be highly appreciated.