2
$\begingroup$

I have two data sets from different collections. The second data set is smaller. They were both analyzed with the same methods in order to derive feature sets of 10-30 features each. Each feature set was produced the same way for both data sets.

Then, I run many Logistic Regressions to fit both data sets with all feature sets. Additionally, all of the experiments were repeated with both L2 regularized and Stepwise Logistic Regression. The observation is that the best fitting for the first data set was done with Stepwise Logistic Regression, while for the second one was done with L2 regularized Logistic Regression. This is quite consistent, i.e. for all 15 experiments on each dataset.

Why did each method perform better on either data-set? May this have to do with particular characteristics of each data-set?

For example, I know that L2 deals better with multicollinearity and lower ratio of observations/variables. Can I assume that L2 performed better on 2nd data set because it had multicollinearity? Then, L2 does not zero out any coefficients, which Stepwise in fact does. Can I say that Stepwise did better on 1st data set because it may have had some too noisy features that needed to be zeroed out?

Improve this question
$\endgroup$
2
  • $\begingroup$ Hi lefterav; your question is quite broad. Are you able to edit to ask more specific questions that could be answered in a few paragraphs? $\endgroup$
    – Glen_b
    Jun 30, 2013 at 0:19
  • $\begingroup$ Does it seem better now? I have seen even more general questions asking for a broad comparison between L1 and L2. I practically ask for a comparison between L2 and stepwise LogReg applicable to my case. $\endgroup$
    – lefterav
    Jun 30, 2013 at 0:51

1 Answer 1

Reset to default
3
$\begingroup$

What you observed follows from statistical theory and is completely expected. And "best fitting on the first dataset" is a by-product of overfitting and is not really interesting.

Why the need to reduce the number of features to 10-30? Why is parsimony good? Why not just fit an L2 penalized model with all features?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Dear Prof Harell, I would appreciate a more clear explanation or pointers to the theory that explain why the observation is completely expected. Best fitting on the first data-set also has the best fit on a cross-validation over all available data, so I am not sure if I can consider this overfitting. We have reduced the number of features to 10-30 because observation shows that (a) when adding more features, results go down (b) when running with all features our classifier script fails with a "not enough examples with so many attributes" error $\endgroup$
    – lefterav
    Jun 30, 2013 at 22:47
  • 1
    $\begingroup$ As a side comment you would have had to reduce features to 10-30 not using $Y$ for your approach to work. Stepwise feature selection without penalization has been proven to perform badly. Penalized maximum likelihood estimation (e.g., L2 shrinkage) intentionally underfits the data at hand so that there is no overfitting of future validation data. Well-fitting the initial dataset can be a bad idea. Consider 9 features on 100 subjects yielding an $R^2$ of 0.4. Randomly removing 90 subjects and refitting will result in an $R^2$ of 1.0 on the training data but near 0.0 on new data. $\endgroup$
    – Frank Harrell
    Jul 1, 2013 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.