# Better Classification of default in logistic regression

Full Disclosure: This is homework. I've included a link to the dataset ( http://www.bertelsen.ca/R/logistic-regression.sav )

My goal is to maximize the prediction of loan defaulters in this data set.

Every model that I have come up with so far, predicts >90% of non-defaulters, but <40% of defaulters making the classification efficiency overall ~80%. So, I wonder if there are interaction effects between the variables? Within a logistic regression, other than testing each possible combination is there a way to identify potential interaction effects? Or alternatively a way to boost the efficiency of classification of defaulters.

I'm stuck, any recommendations would be helpful in your choice of words, R-code or SPSS syntax.

My primary variables are outlined in the following histogram and scatterplot (with the exception of the dichotomous variable)

A description of the primary variables:

age: Age in years
employ: Years with current employer
income: Household income in thousands
debtinc: Debt to income ratio (x100)
creddebt: Credit card debt in thousands
othdebt: Other debt in thousands
default: Previously defaulted (dichotomous, yes/no, 0/1)
ed: Level of education (No HS, HS, Some College, College, Post-grad)


Additional variables are just transformations of the above. I also tried converting a few of the continuous variables into categorical variables and implementing them in the model, no luck there.

If you'd like to pop it into R, quickly, here it is:

## R Code


• Commented Nov 23, 2010 at 17:46
• The raw proportion of defaulters is apparently 1 case out of 4, but it seems you also have a lot of variables. Did you try them all, have you any set of primary variables of interest?
– chl
Commented Nov 23, 2010 at 21:00
• The original variables are at the front of the file. The rest are transformations of the same identified by x_ (where x = log, ln, inv, sqrt). I've tried a mixture of these. But I'm a little bit miffed as to how to interpret or create residual plots where the predictor is 0,1 Commented Nov 23, 2010 at 21:16
• In terms of variables of interest, I've tried all of them primaries and a number of different combinations of the transformed variables as well as mixed models that include interaction effects. Still, nothing higher than 81.7% overall efficiency. Commented Nov 23, 2010 at 21:18

In unbalanced datasets such as this, you can usually improve classification performance by moving away from using a fitted probability of .5 as your cutpoint for classifying cases into defaulters and non-defaulters. For example, I get correct classification rates of .88 and .58 with a cutpoint of .4 for a glm with all 2nd-order interactions. (Which probably leads to overfitting and seems to have some rank issues, but that's another story.)

Code:

m <- glm(default ~ (age + employ + address + income + debtinc +
creddebt + othdebt + ed)^2,
family=binomial(), data=df)
p <- predict(m, newdata=df, type="response")

getMisclass <- function(cutoff, p, labels){
pred <- factor(1*(p > cutoff), labels=c("No Default", "Default"))
t <- table(pred, labels)
cat("cutoff ", cutoff, ":\n")
print(t)
cat("correct    :", round(sum(t[c(1,4)])/sum(t), 2),"\n")
cat("correct No :", round(t[1]/sum(t[,1]), 2),"\n")
cat("correct Yes:", round(t[4]/sum(t[,2]), 2),"\n\n")
invisible(t)
}
cutoffs <- seq(.1,.9,by=.1)
sapply(cutoffs, getMisclass, p=p, labels=df$default)  partial output: cutoff 0.3 : labels pred No Yes No Default 3004 352 Default 740 903 correct : 0.78 correct No : 0.8 correct Yes: 0.72 cutoff 0.4 : labels pred No Yes No Default 3278 532 Default 466 723 correct : 0.8 correct No : 0.88 correct Yes: 0.58 cutoff 0.5 : labels pred No Yes No Default 3493 685 Default 251 570 correct : 0.81 correct No : 0.93 correct Yes: 0.45 cutoff 0.6 : labels pred No Yes No Default 3606 824 Default 138 431 correct : 0.81 correct No : 0.96 correct Yes: 0.34  • Thanks for trying, I played with the threshold as well and it provided little boost to the classification. Commented Nov 24, 2010 at 16:28 • Although, it did do a better job of predicting defaulters. Commented Nov 24, 2010 at 16:52 In logistic regression, highly skewed distributions of outcome variables (where there are far more non-events to events or vis versa), the cut point or probability trigger does need to be adjusted, but it will not have much of an effect on overall classification efficieny. This will always remain roughly the same, but you are currently under-classifying events since the "chance" probability in such a data set will always make you more likely to classify into non-events. This needs to be adjusted for. In fact, in such a situation it's not uncommon to see the overall efficiency of classification go down,since it was previously inflated by misscalculation due to chance. Think of it this way, if you have an event where 90% don't do it and 10% do it, then if you put everyone into the "don't do it" group, you automatically get 90% right, and that was without even trying, just pure chance, inflated by the skewness of it's distribution. The issue of interactions is unrelated to this skewing, and should be driven by theory. You will most likely always improve classification by adding additional terms, including simply adding interactions, but you do so by often overfitting the model. You then have to go back and be able to interpret this. Matt P Data Analyst, University of Illinois Urbana Champaign I'm not a logistic regression expert, but isn't it just a problem of unbalanced data? Probably you have much more non-defaulters than defaulters what may shift the prediction to deal better with larger class. Try to kick out some non-defaulters and see what happens. • I tried it out, presented no substantial increases or decreases in overall efficiency (efficiency being how well it predicted defaulters/non-defaulters in the absence of false-positives,false-negatives) Commented Nov 23, 2010 at 22:16 • @Brandon I've tried few other ideas and it doesn't seem to help. This suggests that this set is just hard enough to make this happen (possibly default is just driven by some unpredictable random factors). – user88 Commented Nov 23, 2010 at 22:32 • @mbq, thanks for taking the time! Much appreciated. Commented Nov 23, 2010 at 22:34 • Personally, I think this is live data that my prof is getting paid to model in one of his consulting jobs... but that's another issue entirely Commented Nov 23, 2010 at 23:38 • the solution to a problem in data analysis should never be "throw away valid data points" - you could try to use balanced training data sets in order to avoid these effects, but you should still evaluate the predictions on all of the data (i.e. all of the validation set). Commented Nov 24, 2010 at 11:46 You might just try including all of the interaction effects. You can then use L1/L2-regularized logistic regression to minimize over-fitting and take advantage of any helpful features. I really like Hastie/Tibshirani's glmnet package (http://cran.r-project.org/web/packages/glmnet/index.html). • The only problem is, I have to be able to reproduce this with SPSS output. :( I'm still going to try it though! Commented Nov 24, 2010 at 4:41 • Tried it out, can't seem to get predict.glmnet() to work. Is there any magic that needs to be happening when you set newx? Commented Nov 24, 2010 at 5:09 I know your question is about logistic regression and as it is a homework assignment so your approach may be constrained. However, if your interest is in interactions and accuracy of classification, it might be interesting to use something like CART to model this. Here's some R code to do produce the basic tree. I've set rpart loose on the enire data frame here. Perhaps not the best approach without some prior knowledge and a cross validation method: library(foreign) df <- read.spss(file="http://www.bertelsen.ca/R/logistic-regression.sav", use.value.labels=T, to.data.frame=T) library(rpart) fit<-rpart(default~.,method="anova",data=df) pfit<- prune(fit, cp= fit$cptable[which.min(fit\$cptable[,"xerror"]),"CP"])

# plot the pruned tree
plot(pfit, uniform=TRUE,
main="Pruned Classification Tree for Loan Default")
text(pfit, use.n=TRUE, all=TRUE, cex=.8)


I'm not sure right off how to produce the classifcation table. It shouldn't be too hard from the predicted values from the model object and the original values. Anyone have any tips here?

• Classification and Regression Trees? That is actually a second part to the assignment. After I've maximized the classification I have to classify based on probability deciles. Commented Nov 24, 2010 at 6:44
• Actually, someone helped me with producing the classification table in this related question: stats.stackexchange.com/questions/4832/… Thanks for the example with R, much appreciated, I found similar instruction on the quick-r website. Although, for this puprpose I'm forced to apply CHAID in SPSS. Commented Nov 24, 2010 at 7:00
• predict method for prediction, table(originalClasses,predictedClasses) for table construction. I tried RF (usually it has accuracy as in overfitted CART, but no overfit) and the result was not much better than glm.
– user88
Commented Nov 24, 2010 at 12:20