I generated random values from a normal distribution(using numpy.random.normal) and threw them into a logistic regression model as X variables(~300 of them). Y is binary( around 51% is 1 ) and is from an actual dataset that I'm working on.
I used sklearn LogisticRegression with l1 penalty and did cross validation on the hyperparameter C using AUC as evaluation metric. What bothers me is the C value that yields the highest AUC wouldn't give me all zero coefficients. Since Xs are not relevant at all, wouldn't a constant model work the best? If that's true, the C selected should be small enough to force all coefficients to zero.
I tried a few runs with different seeds and the highest AUCs were significantly higher than 0.5. And my question is how do I tell if it's a spurious relationship or a truly legitimate model.