# Stepwise versus L2 regularized logistic regression: dataset-specific performance

I have two data sets from different collections. The second data set is smaller. They were both analyzed with the same methods in order to derive feature sets of 10-30 features each. Each feature set was produced the same way for both data sets.

Then, I run many Logistic Regressions to fit both data sets with all feature sets. Additionally, all of the experiments were repeated with both L2 regularized and Stepwise Logistic Regression. The observation is that the best fitting for the first data set was done with Stepwise Logistic Regression, while for the second one was done with L2 regularized Logistic Regression. This is quite consistent, i.e. for all 15 experiments on each dataset.

Why did each method perform better on either data-set? May this have to do with particular characteristics of each data-set?

For example, I know that L2 deals better with multicollinearity and lower ratio of observations/variables. Can I assume that L2 performed better on 2nd data set because it had multicollinearity? Then, L2 does not zero out any coefficients, which Stepwise in fact does. Can I say that Stepwise did better on 1st data set because it may have had some too noisy features that needed to be zeroed out?

• Hi lefterav; your question is quite broad. Are you able to edit to ask more specific questions that could be answered in a few paragraphs? – Glen_b -Reinstate Monica Jun 30 '13 at 0:19
• Does it seem better now? I have seen even more general questions asking for a broad comparison between L1 and L2. I practically ask for a comparison between L2 and stepwise LogReg applicable to my case. – lefterav Jun 30 '13 at 0:51

• As a side comment you would have had to reduce features to 10-30 not using $Y$ for your approach to work. Stepwise feature selection without penalization has been proven to perform badly. Penalized maximum likelihood estimation (e.g., L2 shrinkage) intentionally underfits the data at hand so that there is no overfitting of future validation data. Well-fitting the initial dataset can be a bad idea. Consider 9 features on 100 subjects yielding an $R^2$ of 0.4. Randomly removing 90 subjects and refitting will result in an $R^2$ of 1.0 on the training data but near 0.0 on new data. – Frank Harrell Jul 1 '13 at 11:32