I'm working with a dataset of bank loans, trying to predict which loans are going to default based on some pre-loan-subscription features (for instance, what's the credit grade of the borrower, or the amount of the loan, or the borrower's annual income...).

There are roughly 800.000 data points (one for each loan) and about $7\%$ of them are in default (which is a boolean True/False value).

I'm using regression algorithms to output a probability of default for each loan. For instance, given its features values, a particular loan will be assigned a probability of $p = 0.234$ to default. This works pretty well because, for instance, of all loans whose predictions satisfy $p \in [0.20, 0.30[$ in the test set, roughly $25\%$ of them are effectively in default, and so on.

The problem I'm facing is to find a suitable error function to estimate the algorithms' accuracy. Currently I'm using mean absolute error (MAE) which has the following issue: as most loans were not defaulted, if I arbitrarily assign a probability of $p = 0.0$ for every loan, then the mean absolute error will end up being very low, because obviously $p = 0.0$ will be the perfect prediction value for $93\%$ of loans.

What would then be a suitable error function so that its lowest is not when all predictions are arbitrarily set as low as possible, but accurately reflects the quality of the predictions?

To give an illustration of the algorithms output, here are some results, showing the amount of loans predicted for each estimated probability range, and the corresponding proportion of those who have effectively defaulted.

enter image description here

  • 1
    $\begingroup$ See the answer here, it should be helpful: stats.stackexchange.com/questions/222558/… $\endgroup$ Commented Jan 12, 2017 at 11:53
  • 1
    $\begingroup$ BTW, you shouldn't use salary as a predictor, because you don't want to model the "capability", but the "intention" of the customer. $\endgroup$ Commented Jan 12, 2017 at 11:54
  • 1
    $\begingroup$ @UjjwalKumar Yet salary and intention are most likely linked. Therefore salary is relevant input $\endgroup$ Commented Jan 12, 2017 at 12:15
  • $\begingroup$ @UjjwalKumar the link is indeed related - however it concerns classification exclusively - I'm using regression and therefore can't apply the recommendations of the linked answer $\endgroup$
    – Jivan
    Commented Jan 12, 2017 at 12:57
  • 1
    $\begingroup$ Refer to the part of answers where they talk about AUC, and TPR, Recall at different quantiles of probability-scores. Ideally your high probability cases should contains all defaulters, and as you go lower, more and more non-defaulters would creep-in. A model which separates TP and TN perfectly has an AUC of 1, any time a TN appears higher in probability than a TP, AUC falls. $\endgroup$ Commented Jan 12, 2017 at 13:12

1 Answer 1


Strictly proper scoring rules are optimized in expected value by the true probabilities.

More informally, optimizing strictly proper scoring rules leads to your model seeking out the true probabilities. This sounds like the exact goal you have: to make probabilistic predictions as close to the true default probabilities as your can.

Two common strictly proper scoring rules are log loss and Brier score. Below, $y=(y_1,\dots,y_n)$ denotes the true observations $(y_i\in\{0, 1\})$, and $p = (p_1,\dots,p_n)$ denotes the predicted probabilities from a model $(p_i\in[0, 1])$.

$$ \text{Log Loss}\\ L(y, p) = -\dfrac{1}{n}\overset{n}{\underset{i=1}{\sum}}\bigg( y_i\log(p_i) + (1-y_i)\log(1 - p_i) \bigg)\\ \text{Brier Score}\\ B(y, p) = \dfrac{1}{n}\overset{n}{\underset{i=1}{\sum}}\bigg( y_i - p_i \bigg)^2 $$

A complaint about either of these might be that there is no sense of how good a score is, while a typical regression metric of $R^2$ gives a sense of model quality. First, that interpretation of $R^2$ is not as easy as one might hope, so I challenge the idea that any performance metric can be evaluated as "good" or "bad" without a context the way one might like to consider $90\%$ an $\text{A}$ in school while $40\%$ is an $\text{F}$. Second, both Brier score and log loss can be transformed to give $R^2$-style measures of performance. Let $\bar y$ be the proportion of $1$s in your data. Then McFadden's and Efron's pseudo $R^2$ values are defined as follows:

\overset{n}{\underset{i=1}{\sum}}\bigg( y_i\log(p_i) + (1-y_i)\log(1 - p_i) \bigg)

$$ R^2_{\text{McFadden}} = 1 - \left(\dfrac{ \overset{n}{\underset{i=1}{\sum}}\bigg( y_i\log(p_i) + (1-y_i)\log(1 - p_i) \bigg) }{ \overset{n}{\underset{i=1}{\sum}}\bigg( y_i\log(\bar y) + (1-y_i)\log(1 - \bar y) \bigg) }\right) \\ R^2_{\text{Efron}} = 1 - \left(\dfrac{ \overset{n}{\underset{i=1}{\sum}}\bigg( y_i - p_i \bigg)^2 }{ \overset{n}{\underset{i=1}{\sum}}\bigg( y_i - \bar y \bigg)^2 }\right) $$

As the $\bar y$ can be seen as the prior probability of a default, these pseudo $R^2$ values can be seen as comparisons of the values of strictly proper scoring rules achieved by the model compared to the value of that same strictly proper scoring rule that is achieved by predicting the prior probability every time, which is analogous to how the usual $R^2$ can be seen as comparing model predictions of the conditional mean to a model predicting the conditional mean as the marginal mean every time. Think about it this way: if you knew nothing about how your features influenced default probability, what would be your best guess for someone's default default probability if you knew the overall default rate to be $\bar y?$ These pseudo $R^2$ calculations go with the logic that the best naïve approach is to predict $\bar y$ every time.

If you want to do out-of-sample assessments of performance, I have a strong opinion about how to do that, a stance that now has support in the statistics literature from Hawinkel & Waegeman (2023)!

Finally, if you are checking that loans predicted to default with probabilities between $20\%$ and $30\%$ do default with the claimed probability, you are assessing your model calibration. I think the sklearn documentation gives a good idea of what this means and gives some references. What you are doing by binning into an interval like $[0.2, 0.3]$ is evocative of the Hosmer-Lemeshow test, which Frank Harrell argues is obsolete and replaced by techniques like those present in the sklearn function or his rms package (such as rms:calibrate).

I have a few posts on here about probability calibration.

Probability Calibration of Statistical Models

Should I use statistical tests (e.g., Hosmer-Lemeshow) to assess predictive models?

classification ML model: probability of positive label knowing the model score

Walk through rms::calibrate for logistic regression

Walk through rms::val.prob


Hawinkel, Stijn, Willem Waegeman, and Steven Maere. "Out-of-sample R 2: estimation and inference." The American Statistician just-accepted (2023): 1-16.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.