10
$\begingroup$

I am running an analysis on the probability of loan default using logistic regression and random forests.

When I use logistic regression, the prediction is always all '1' (which means good loan). I have never seen this before, and do not know where to start in terms of trying to sort out the issue. There are 22 columns with 600K rows. When I decrease the # of columns I get the same result with logistic regression.

Why could the logistic regression be so wrong?

**Actual from the data**

0 :   41932

1 :   573426

**Logistic regression output** 

prediction for 1 when actually 0: 41932
prediction for 1 when actually 1:573426

A**s you can see, it always predicts a 1**


**Random forests does better:**

actual 0, pred 0 : 38800 
actual 1, pred 0 : 27 
actual 0, pred 1 : 3132
actual 1, pred 1 : 573399
Improve this question
$\endgroup$
6
  • 5
    $\begingroup$ This doesn't make a sense. Logit will not predict exactly 0. It may predict a low value which you interpreted as 0. So, the problem could be due to the threshold, not just the model itself $\endgroup$
    – Aksakal
    Commented Aug 26, 2015 at 20:37
  • $\begingroup$ @Aksakal, I am using the scikit learn .predict method. predict class labels for samples in X $\endgroup$
    – ivan7707
    Commented Aug 31, 2015 at 19:43
  • $\begingroup$ Are you familiar with ROC curves? You can extract the predicted probabilities, then play with the threshold to classify the data yourself. The threshold is your trade-off lever between identifying either defaults or non-defaults. $\endgroup$
    – Aksakal
    Commented Aug 31, 2015 at 19:48
  • 1
    $\begingroup$ See my answer below, but also you can use ROC to find the sweet spot in your classifier setting for logit between sensitivity and specificity $\endgroup$
    – Aksakal
    Commented Aug 31, 2015 at 20:15
  • 4
    $\begingroup$ Dont use predict in sklearn on a probability model, it's useless. ALWAYS use predict_proba. $\endgroup$
    – Matthew Drury
    Commented Sep 7, 2016 at 21:42

6 Answers 6

Reset to default
5
$\begingroup$

Well, it does make sense that your model predicts always 1. Have a look at your data set: it is severly imbalanced in favor of your positive class. The negative class makes up only ~7% of your data. Try re-balancing your training set or use a cost-sensitive algorithm.

Improve this answer
$\endgroup$
5
  • $\begingroup$ thanks for the input. Is there a rule of thumb for what is acceptable for unbalanced data, or good sources for how to re-balance that you could suggest? $\endgroup$
    – ivan7707
    Commented Aug 26, 2015 at 20:48
  • $\begingroup$ Unfortunately, there is no rule on to how to pick an algorithm but the "no free lunch theorem". In your particular case I would go with Ross Quinlan's C5.0 package, first. Then you could experiment with different costs and sampling techniques like up- and downsampling, SMOTE etc. In addition, Max Kuhn's site offers a nice summary of established algorithms. $\endgroup$
    – JimBoy
    Commented Aug 26, 2015 at 21:09
  • 7
    $\begingroup$ (+1) In the absence of a cost function there seems to be no reason to use logistic regression as a classifier: you have the predicted probabilities & can use a proper scoring rule to assess your model's performance. See e.g.What's the measure to assess the binary classification accuracy for imbalanced data?. Imbalance is not a problem per se: see Does down-sampling change logistic regression coefficients?. $\endgroup$
    – Scortchi ♦
    Commented Aug 27, 2015 at 16:04
  • $\begingroup$ @Scortchi, thanks for the links and the idea of using models with costs. I was able to find this paper link which gets me going in the right direction. $\endgroup$
    – ivan7707
    Commented Aug 31, 2015 at 16:29
  • 1
    $\begingroup$ No, it doesn't make a sense that his model predicts always 1s, because 7% is a rather high rate of default and logit is used widely in loan defaults. Consider AAA rated loans which default at 0.1% annually. His are basically junk loans. $\endgroup$
    – Aksakal
    Commented Aug 31, 2015 at 19:49
16
$\begingroup$

The short answer is that logistic regression is for estimating probabilities, nothing more or less. You can estimate probabilities no matter how imbalanced $Y$ is. ROC curves and some of the other measures given in the discussion don't help. If you need to make a decision or take an action you apply the loss/utility/cost function to the predicted risk and choose the action that optimizes the expected utility. It seems that a lot of machine learning users are not really understanding risks and optimum decisions.

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ (+1) Yes, the question is "are you solving a classification problem, or are you solving a decision-support problem?". $\endgroup$
    – GeoMatt22
    Commented Sep 8, 2016 at 14:54
  • 1
    $\begingroup$ I'm uncertain about that. Estimation of probabilities is a great end result. And note that the majority of "classification" problems are better addressed using optimal Bayes decisions. Other than visual and audio pattern recognition, most problems where classification methods are applied would be better addressed with direct probability estimation. $\endgroup$
    – Frank Harrell
    Commented Sep 8, 2016 at 15:17
  • $\begingroup$ @FrankHarrell Is it correct that interpreting the output as probabilities requires a design that allows such an interpretation (cohort). And if we don't have such a design then we have to make a decision based on the "risk scores". Further, although there is literature discussing this in the non-calibrated setting, this is not that common in practice. Is this correct? $\endgroup$
    – julieth
    Commented Sep 12, 2016 at 2:31
  • 1
    $\begingroup$ Please describe how the sampling used to assemble the dataset used for model development differs from the customers to whom you will apply the predictions. $\endgroup$
    – Frank Harrell
    Commented Sep 12, 2016 at 12:17
  • $\begingroup$ For example, case-control sampling for which target prevalence is unknown. Or moderately-sized convenience samples. $\endgroup$
    – julieth
    Commented Sep 13, 2016 at 1:06
3
$\begingroup$

If the problem is indeed the imbalance between the classes, I would simply start by balancing the class weights: 

log_reg = LogisticRegression(class_weight = 'balanced')

This parameter setting means that the penalties for false predictions in the loss function will be weighted with inverse proportions to the frequencies of the classes. This can solve the problem you describe.

Improve this answer
$\endgroup$
3
  • $\begingroup$ It is not clear to me that you have pinpointed the problem. I think Matthew Drury hit on the problem which had to do with the use of sklearn. $\endgroup$
    – Michael R. Chernick
    Commented Nov 21, 2017 at 20:34
  • $\begingroup$ Michael may be right, but this did solve my problem and now my model is predicting 1's when it was not before! $\endgroup$
    – embulldogs99
    Commented Jun 10, 2021 at 21:48
  • $\begingroup$ @embulldogs99, My model has also started predicting both classes now, but the accuracy is reduced to even lower than when all the observations were assigned to the same class! $\endgroup$
    – Martund
    Commented Sep 3, 2021 at 5:28
2
$\begingroup$

When you classify using logit, this is what happens.

The logit predicts the probability of default (PD) of a loan, which is a number between 0 and 1. Next, you set a threshold D, such that you mark a loan to default if PD>D, and mark it as non-default if PD

Naturally, in a typical loan population PD<<1. So, in your case 7% is rather high probability of it's one year data (PDs are normally reported on annual basis). If this is multi year data, then we're talking about so called cumulative PD, in this case cumPD=7% is not a high number for 10 years of data, for instance. Hence, by any standards, I wouldn't say that your data set is problematic. I'd describe it at least typical for loan default data, if not great (in the sense that you have relative large number of defaults).

Now, suppose that your model predicting the following three levels of PD:

  • 0.1 (563,426)
  • 0.5 (20,000)
  • 0.9 (31,932)

Suppose also that the actual defaults for these groups were:

  • 0
  • 10,000
  • 31,932

Now you can set D to different values and see how the matrix changes. Let's use D = 0.4 first:

  • Actual default, predict non-default: 0
  • Actual default, predict default: 41,932
  • Actual non-default, predict non-default: 563,426
  • Actual non-default, predict default: 10,000

If you set D = 0.6:

  • Actual default, predict non-default: 31,932
  • Actual default, predict default: 10,000
  • Actual non-default, predict non-default: 573,426
  • Actual non-default, predict default: 0

If you set D = 0.99:

  • Actual default, predict non-default: 41,932
  • Actual default, predict default: 0
  • Actual non-default, predict non-default: 573,426
  • Actual non-default, predict default: 0

The last case is what you see in your model results. In this case I'm emphasizing the threshold D for a classifier. A simple in change in D may improve certain characteristics of your forecast. Note, that in all three cases the predicted PD remained the same, only the threshold D has changed.

It is also possible that your logit regression itself is crappy, of course. So, in this case you have at least two variables: the logit spec and the threshold. Both impact your forecast power.

Improve this answer
$\endgroup$
11
  • $\begingroup$ You do realize that your'e proposing a technique to deal with imbalanced data, do you? Therefore, you're admitting the effect of the smaller class on the prediction accuracy. In addition, you're proposing a technique that the original model isn't using at all. You can't just change the circumstances to your liking and then make up some statement as you go along. $\endgroup$
    – JimBoy
    Commented Aug 31, 2015 at 20:30
  • $\begingroup$ In loan default analysis/forecasting the data is always "imbalanced" in this sense. It's the normal state of affairs. $\endgroup$
    – Aksakal
    Commented Aug 31, 2015 at 20:46
  • $\begingroup$ This may be as it is. Nonetheless, you should have a look at what Max Kuhn describes as the "no information rate", which is nothing else than the largest class in the data set. So, have a look at the table Ivan provided again. The results make perfect sense for the model he used. That you can actually optimize those results with different techniques is another question and entirely possible. $\endgroup$
    – JimBoy
    Commented Aug 31, 2015 at 21:24
  • $\begingroup$ @JimBoy, I saw his table, and seen many more like that. His is rather simple, we usually deal with loan delinquency data, where the states are all the way from Current to 30 days past due, 60, 90 .... through Default and Closed. In a good portfolio you can have 95% loans in Current (clean) state, and only 1% in Default. People use mulltinomial logit for this kind of stuff all the time in the industry. $\endgroup$
    – Aksakal
    Commented Aug 31, 2015 at 21:31
  • $\begingroup$ @Aksakal, I'll have to do more reading on changing the threshold, as I have read a lot about how it is mathematically incorrect to change it for logistic regression. On another note, what did you mean by 'it is possible that your logit regression itself is crappy'? $\endgroup$
    – ivan7707
    Commented Sep 1, 2015 at 15:23
0
$\begingroup$

Well, without more information its hard to say, but by the definition of logistic regression you are saturating based on the fitted data. So in the equation the e^-t term is going to 0. So the first place to look would be to see what the actual coefficients are.

This could also be due to poorly scaled variables. There might be an issue where one of the columns is huge in numerical value compared to others that is causing it mess up.

Improve this answer
$\endgroup$
1
  • $\begingroup$ @ Tim Felty, Thanks for the response. Can you please expand on what I would be looking for regarding the coefficients and how this relates to saturation (or point me to a resource to read)? Also, I was under the impression that poorly scaled variables would not have a negative effect on the logistic regression. [link(]stats.stackexchange.com/questions/18916/… ) $\endgroup$
    – ivan7707
    Commented Aug 26, 2015 at 20:34
0
$\begingroup$

You may use SMOTE to balance the unbalanced dataset. A good paper for reference is:

Lifeng Zhou, Hong Wang, Loan Default Prediction on Large Imbalanced Data Using Random Forests, TELKOMNIKA Indonesian Journal of Electrical Engineering, Vol.10, No.6, October 2012, pp. 1519~1525, link.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Could you add a full citation/reference (including author, date, publisher etc) as you would in an academic paper? This would make it easier for future readers to track it down if the link stopped working $\endgroup$
    – Silverfish
    Commented Sep 7, 2016 at 22:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.