I have a binary response Y and a n*p X matrix as predictors, which n = 100 and p = 100000. I use logistic regression function to build the classification model. $ \log \frac{p(Y=1|X)}{Y=0|X} = \beta_0 + \beta_iX$. I want to check the predictive performance of MLE in the high-dimensional setting. I use the cross-validation method with the AUC value as prediction criterion to check the predictive performance of MLE. Through the Glmnet function on training data, I obtain the MLE for each predictor X, then predict the test data set based on the MLEs. However, I got a very high AUC value. In my interpretation, the MLE should perform badly in high-dimensional setting, because the MLEs are not unique, the MLEs may be overestimated. Could anyone explain to me whether it is reasonable that the MLE perform very good in a high-dimensional setting? Why can it be that?

  • $\begingroup$ You should detail better your question, e.g. what kind of resampling (if any) are you using to obtain that AUC? Are you really sure your AUC computation is right? $\endgroup$ – Firebug Sep 1 '17 at 19:59
  • $\begingroup$ @Firebug I have added some more details to my question. Can you check again? $\endgroup$ – user5802211 Sep 1 '17 at 20:26
  • 1
    $\begingroup$ Glmnet does not do maximum likelihood estimation. It penalizes the likelihood with a measure of model complexity, ie, regularization. $\endgroup$ – Matthew Drury Sep 2 '17 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.