1
$\begingroup$

I generated random values from a normal distribution(using numpy.random.normal) and threw them into a logistic regression model as X variables(~300 of them). Y is binary( around 51% is 1 ) and is from an actual dataset that I'm working on.

I used sklearn LogisticRegression with l1 penalty and did cross validation on the hyperparameter C using AUC as evaluation metric. What bothers me is the C value that yields the highest AUC wouldn't give me all zero coefficients. Since Xs are not relevant at all, wouldn't a constant model work the best? If that's true, the C selected should be small enough to force all coefficients to zero.

I tried a few runs with different seeds and the highest AUCs were significantly higher than 0.5. And my question is how do I tell if it's a spurious relationship or a truly legitimate model.

Thank you!

$\endgroup$
1
  • $\begingroup$ There is a very long history, and a multitude of posts on this site about overfitting and multiple comparison problems. It would be good to survey these areas first. $\endgroup$ – Frank Harrell Dec 20 '20 at 12:56
0
$\begingroup$

Update on this in case anyone has the same question. The sample size was not big enough. After I increased the sample size, hyperparameter C selected would always be small enough to keep the coefficients at 0.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.