1
$\begingroup$

I am analyzing the probability of default on loans with the aid of a binary logistic regression model. I have therefore fitted the logit of default: $log\dfrac{P(Y=1|\boldsymbol{x})}{1-P(Y=1|\boldsymbol{x})} = \boldsymbol{X\beta}$.

edited

The goal of the analysis is set up a model that can be used to classify borrowers in categories depending on the risk of defaulting. I want to use this model when deciding on setting an appropriate interest rate and also declining high risk borrowers loans.

Clearly my model needs to be able to predict/forecast future risk so it has to be well calibrated.

As a measure of performance, I am unsure whether I can use AUROC in any aspect? As for my understanding is that diagnosis can be made att present time; but I can't tell you anything about the borrower in present time(default or not). Therefore I don't see why my model should be able to discriminate at all.

Does anyone agree or disagree with the fact that I don't need to consider the discriminatory power of my model? can you give my an eyeopener?

Iam trying to understand it with an example:

Suppose I have 2 competing models $A$ with AUC $100$% and $B$ with AUC $50$%

For some subjects of $A$ we predict predict probability of default $s_1 = 0.6, s_2=0.61,s_3 = 0.4,s_4=0.55,s_5 = 0.3,s_5 = 0.2, s_6=0.6,s_7 = 0.59,s_8=0.7,s_9 = 0.1, s_{10} = 0.62$

Say we have some cutoff $0.5$ so that we reject loans with PD $>0.5$. Then we would reject $s_1,s_2,s_4,s_6,s_7,s_8,s_{10}$

why is $A$ preferable to by $B$? it is still the the case that all the rejected $\{s_j\}$ could default, so why is model $A$ better?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Your conceptualization has a lot of problems, and this is symbolized by the use in inappropriate terminology, e.g., using the word multivariate when you have only a univariate $Y$. The goal of credit risk scoring is to estimate credit risk, not to create categories for borrowers. Risk is a continuous measure that should be used continuously (look up lift curves as an example).

Then you confused longitudinal with cross-sectional designs. Most (but not all) risk analyses use cross-sectional data, then envision applying the predictions to a future, similar cohort of subjects. For that setup, measuring predictive discrimination ($c$ index aka AUROC) is a good idea. Rank-based indexes such as $c$ are not optimal (although they are useful and more understandable) when compared to likelihood-based indexes such as pseudo $R^2$.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your answer Frank. How much attention should I pay to make agreement between observed outcomes and predictions in a cross-sectional designs? does it matter? @Frank Harell $\endgroup$
    – Danny
    Oct 14, 2017 at 13:25
  • $\begingroup$ You should pay a lot of attention to it but not by thinking of agreement. You need an unbiased smooth estimate of the whole calibration curve to show absolute accuracy of estimated risks. $\endgroup$
    – Frank Harrell
    Oct 14, 2017 at 13:44
  • $\begingroup$ if I have a model that discriminates well but is not well calibrated what implications does that have in terms of risk analysis? If I have well calibrated model but the discriminatory power is poor what does that imply in terms of risk analysis. Finally if good at both what does that imply in terms of risk analysis @Frank Harrell $\endgroup$
    – Danny
    Oct 14, 2017 at 14:00
  • $\begingroup$ Very long literature about this. Take a look. Briefly: discrimination is judged by comparing to competing models; low discrim = not misleading but not as useful. Absolute calibration is essential unless just doing a lift curve (i.e., just ranking customers). You want a predicted risk of 0.2 not to turn out to be 0.4. $\endgroup$
    – Frank Harrell
    Oct 14, 2017 at 16:40
  • $\begingroup$ Really thanks for taking time to answer. I am still struggling to understand but I complemented with an example, if you could maybe say something about it and I could get closer to understand this issue @Frank Harrell $\endgroup$
    – Danny
    Oct 15, 2017 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.