4
$\begingroup$

I'm trying to build a logistic regression model to predict 90+ Days past due(DPD) events. The size of the database is 96000, with an event rate of 6%. We ran the entire data set through the info value process, and converted it into Weight of Evidence (WOE) bins. When I try to build the model using 60% of the data (for development data, the other 40% are held out for validation), the logistic regression model gives me 7 significant variables with a very high wald score for the intercept. Below, I give the results from the model:

  1. The overall model is significant (P<.0001).
  2. ROC 0.84.
  3. The Hosmer and Lemeshow Test is significant (P<0.0001), which implies the model does not adequately fit the data.
  4. The accuracy of this model is poor, and has a correct classification rate of 21%.

Please tell me your views on this, specifically: . Are there any ways/methods that can help improve performance on the HL Test, since we need use the probability for the prediction? . Can I ignore a few good loans based on some business rules? . Is there a different methodology we should try?

I'm fairly new to the credit risk modeling and looking forward for your view.

Thank you in advance

Improve this question
$\endgroup$
2
  • 6
    $\begingroup$ Your question might be clearer if you defined your abbreviated terms! ROC is common knowledge, but I doubt DPD and WOE are. $\endgroup$
    – Kyle.
    Nov 20, 2012 at 19:28
  • 4
    $\begingroup$ Statistical significance is not a good indicator of the value of a model - with such high N, even very small effects will be significant $\endgroup$
    – Peter Flom
    Nov 20, 2012 at 19:49

2 Answers 2

Reset to default
1
$\begingroup$

You can adjust the penalty for different sorts of wrong classification. Consider that you could get 94% correct just by putting everyone in the "no event" category. But that wouldn't be useful. Clearly, whatever program you are using is, implicitly or explicitly, using a different weighting.

You can look at graphs: For continuous independent variables you can look at parallel boxplots. For categorical ones you can look at mosaic plots.

You can avoid binning, which can lower statistical power.

You could try other methods, such as classification trees.

But, in the end, you may just not have a good model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you Peter. As you had mentioned i will try with 2 options. 1) Excluding the bins with lower statistical power but this untimely i will end up with some data loss. 2)Try with the Classification trees. $\endgroup$
    – Gopi
    Nov 21, 2012 at 4:51
0
$\begingroup$

Disclaimer: I'm new to risk scoring too. And I know this thread is 6 yrs. old.

Classification is one thing. In LR you can simply adjust the cut-off (the probability above which to classify to target event). The goodness-of-fit tests are different thing. That is how well your new distributions fits the theoretical (model) one. On a practical note, make sure you have equally distributed train-test sets and you don't have strongly correlated factors (independent vars).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Note that LR is not a common abbreviation for "logistic regression", since it is more often used as "likelihood ratio" by statisticians $\endgroup$
    – runr
    Jun 13, 2019 at 12:47
  • $\begingroup$ Noted. However, the context above should be sufficient. $\endgroup$
    – Julian
    Jun 13, 2019 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.