2
$\begingroup$

I did a Logistic Regression (LR) on a 2-class problem (77.3% negative, 22.7% positive), and the results are as follow:

$\text{logit} (p) = -2.0 + 1.4X_1 + 1.3X_2 + 0.2X_3 - 0.3X_4 - 0.7X_5$

The final model Likelihood Ratio Tests indicated the model is significantly better than the intercept-only model (Chi-square = 21.636, df=5, p = 0.001).

The goodness-of-fit is also indicating the model is good.

Pearson Chi-square = 12.777, df=12, p = 0.385

Deviance Chi-square = 16.007, df=12, p = 0.191

Pseudo R-Square

Cox and Snell .033

Nagelkerke .050

McFadden .031

Everything seems ok, but my classification result is not so good. Basically, it classifies everything as negative (Model accuracy = 77.3% same as the baseline).

My question is whether the estimated coefficient from this model is still any good considering its classification performance is not.

I added the ROC of the positive class.

ROC of the positive class

$\endgroup$
  • 4
    $\begingroup$ Please explain how this model "classifies" things. Logistic regression ordinarily lets you estimate probabilities of responses, but it does not involve any form of classification--that's something you add on to the results and there are many different ways to do that. $\endgroup$ – whuber Mar 29 '17 at 15:05
  • $\begingroup$ I used a cut-off point at 0.5. $\endgroup$ – user1480478 Mar 29 '17 at 15:06
  • $\begingroup$ From the limited degrees of freedom, it seems that you only have a handful of cases and probably less than 10 in the least-prevalent class. If so you are severely over-fitting; you should have 10-20 of the least-prevalent cases for each predictor variable you are examining, or 50-100 for your model. $\endgroup$ – EdM Mar 29 '17 at 17:02
9
$\begingroup$

You have chosen too large a cutoff: 0.5 is much greater than the prevalence of the outcome (0.23) so you have 100% specificity and 0% sensitivity. For the classification ability of logistic model, you should consider all possible cut-offs in a receiver operating characteristic curve and choose a cutoff for optimal sensitivity/specificity.

$\endgroup$
  • $\begingroup$ Should LG automatically take care of this (imbalanced label)? where optimizing logistic loss, the optimal solution is using 0.5 cut off $\endgroup$ – Haitao Du Mar 29 '17 at 15:52
  • 1
    $\begingroup$ I dont believe that can be correct hxd. A cutoff must be chosen with respect to some cost function. A logistic regression itself does not care about threasholding, nor should it. This is the same point whuber made above. $\endgroup$ – Matthew Drury Mar 29 '17 at 16:02
  • $\begingroup$ @hxd1011 no. Logistic regression models the probability of being included as a case in the sample. The average predicted probabilities will always equal the proportion of cases in the sample. $\endgroup$ – AdamO Mar 29 '17 at 16:06
  • 1
    $\begingroup$ Yup, pretty much. The job of the regression is to estimate conditional probabilities, which is the correct goal for a statistical or machine learning model as it maximizes flexibility. It's your job as a scientist to, once you have probability estimates, take classification action according to your scientific or business goals. @user1480478 As a consequence, since the regressions job is to accurately estimate conditional probabilities, the coefficient estimates are correct (conditional on the model fitting the data well). $\endgroup$ – Matthew Drury Mar 29 '17 at 16:44
  • 1
    $\begingroup$ @user1480478 jointly or individually? And what constitutes "good" in your opinion? Don't take for granted the many applications of statistical modeling as if it's universally known what you're trying to do... $\endgroup$ – AdamO Mar 29 '17 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.