13
$\begingroup$

Full Disclosure: This is homework. I've included a link to the dataset ( http://www.bertelsen.ca/R/logistic-regression.sav )

My goal is to maximize the prediction of loan defaulters in this data set.

Every model that I have come up with so far, predicts >90% of non-defaulters, but <40% of defaulters making the classification efficiency overall ~80%. So, I wonder if there are interaction effects between the variables? Within a logistic regression, other than testing each possible combination is there a way to identify potential interaction effects? Or alternatively a way to boost the efficiency of classification of defaulters.

I'm stuck, any recommendations would be helpful in your choice of words, R-code or SPSS syntax.

My primary variables are outlined in the following histogram and scatterplot (with the exception of the dichotomous variable)

A description of the primary variables:

age: Age in years
employ: Years with current employer
address: Years at current address
income: Household income in thousands
debtinc: Debt to income ratio (x100)
creddebt: Credit card debt in thousands
othdebt: Other debt in thousands
default: Previously defaulted (dichotomous, yes/no, 0/1)
ed: Level of education (No HS, HS, Some College, College, Post-grad)

Additional variables are just transformations of the above. I also tried converting a few of the continuous variables into categorical variables and implementing them in the model, no luck there.

If you'd like to pop it into R, quickly, here it is:

## R Code
df <- read.spss(file="http://www.bertelsen.ca/R/logistic-regression.sav", use.value.labels=T, to.data.frame=T)

alt text alt text

$\endgroup$
4
  • $\begingroup$ Related: stats.stackexchange.com/questions/4831/… and; stats.stackexchange.com/questions/4832/… $\endgroup$ Nov 23, 2010 at 17:46
  • $\begingroup$ The raw proportion of defaulters is apparently 1 case out of 4, but it seems you also have a lot of variables. Did you try them all, have you any set of primary variables of interest? $\endgroup$
    – chl
    Nov 23, 2010 at 21:00
  • $\begingroup$ The original variables are at the front of the file. The rest are transformations of the same identified by x_ (where x = log, ln, inv, sqrt). I've tried a mixture of these. But I'm a little bit miffed as to how to interpret or create residual plots where the predictor is 0,1 $\endgroup$ Nov 23, 2010 at 21:16
  • $\begingroup$ In terms of variables of interest, I've tried all of them primaries and a number of different combinations of the transformed variables as well as mixed models that include interaction effects. Still, nothing higher than 81.7% overall efficiency. $\endgroup$ Nov 23, 2010 at 21:18

5 Answers 5

8
$\begingroup$

In unbalanced datasets such as this, you can usually improve classification performance by moving away from using a fitted probability of .5 as your cutpoint for classifying cases into defaulters and non-defaulters. For example, I get correct classification rates of .88 and .58 with a cutpoint of .4 for a glm with all 2nd-order interactions. (Which probably leads to overfitting and seems to have some rank issues, but that's another story.)

Code:

m <- glm(default ~ (age + employ + address + income + debtinc + 
                    creddebt + othdebt + ed)^2,
   family=binomial(), data=df)
p <- predict(m, newdata=df, type="response")

getMisclass <- function(cutoff, p, labels){
   pred <- factor(1*(p > cutoff), labels=c("No Default", "Default")) 
   t <- table(pred, labels)
   cat("cutoff ", cutoff, ":\n")
   print(t)
   cat("correct    :", round(sum(t[c(1,4)])/sum(t), 2),"\n")
   cat("correct No :", round(t[1]/sum(t[,1]), 2),"\n")
   cat("correct Yes:", round(t[4]/sum(t[,2]), 2),"\n\n")
   invisible(t)
}
cutoffs <- seq(.1,.9,by=.1)
sapply(cutoffs, getMisclass, p=p, labels=df$default)

partial output:

cutoff  0.3 :
            labels
pred           No  Yes
  No Default 3004  352
  Default     740  903
correct    : 0.78 
correct No : 0.8 
correct Yes: 0.72 

cutoff  0.4 :
            labels
pred           No  Yes
  No Default 3278  532
  Default     466  723
correct    : 0.8 
correct No : 0.88 
correct Yes: 0.58 

cutoff  0.5 :
        labels
pred           No  Yes
  No Default 3493  685
  Default     251  570
correct    : 0.81 
correct No : 0.93 
correct Yes: 0.45 

cutoff  0.6 :
            labels
pred           No  Yes
  No Default 3606  824
  Default     138  431
correct    : 0.81 
correct No : 0.96 
correct Yes: 0.34 
$\endgroup$
2
  • $\begingroup$ Thanks for trying, I played with the threshold as well and it provided little boost to the classification. $\endgroup$ Nov 24, 2010 at 16:28
  • 1
    $\begingroup$ Although, it did do a better job of predicting defaulters. $\endgroup$ Nov 24, 2010 at 16:52
5
$\begingroup$

In logistic regression, highly skewed distributions of outcome variables (where there are far more non-events to events or vis versa), the cut point or probability trigger does need to be adjusted, but it will not have much of an effect on overall classification efficieny. This will always remain roughly the same, but you are currently under-classifying events since the "chance" probability in such a data set will always make you more likely to classify into non-events. This needs to be adjusted for. In fact, in such a situation it's not uncommon to see the overall efficiency of classification go down,since it was previously inflated by misscalculation due to chance.

Think of it this way, if you have an event where 90% don't do it and 10% do it, then if you put everyone into the "don't do it" group, you automatically get 90% right, and that was without even trying, just pure chance, inflated by the skewness of it's distribution.

The issue of interactions is unrelated to this skewing, and should be driven by theory. You will most likely always improve classification by adding additional terms, including simply adding interactions, but you do so by often overfitting the model. You then have to go back and be able to interpret this.

Matt P Data Analyst, University of Illinois Urbana Champaign

$\endgroup$
4
$\begingroup$

I'm not a logistic regression expert, but isn't it just a problem of unbalanced data? Probably you have much more non-defaulters than defaulters what may shift the prediction to deal better with larger class. Try to kick out some non-defaulters and see what happens.

$\endgroup$
6
  • $\begingroup$ I tried it out, presented no substantial increases or decreases in overall efficiency (efficiency being how well it predicted defaulters/non-defaulters in the absence of false-positives,false-negatives) $\endgroup$ Nov 23, 2010 at 22:16
  • 1
    $\begingroup$ @Brandon I've tried few other ideas and it doesn't seem to help. This suggests that this set is just hard enough to make this happen (possibly default is just driven by some unpredictable random factors). $\endgroup$
    – user88
    Nov 23, 2010 at 22:32
  • $\begingroup$ @mbq, thanks for taking the time! Much appreciated. $\endgroup$ Nov 23, 2010 at 22:34
  • 1
    $\begingroup$ Personally, I think this is live data that my prof is getting paid to model in one of his consulting jobs... but that's another issue entirely $\endgroup$ Nov 23, 2010 at 23:38
  • 1
    $\begingroup$ the solution to a problem in data analysis should never be "throw away valid data points" - you could try to use balanced training data sets in order to avoid these effects, but you should still evaluate the predictions on all of the data (i.e. all of the validation set). $\endgroup$
    – fabians
    Nov 24, 2010 at 11:46
2
$\begingroup$

You might just try including all of the interaction effects. You can then use L1/L2-regularized logistic regression to minimize over-fitting and take advantage of any helpful features. I really like Hastie/Tibshirani's glmnet package (http://cran.r-project.org/web/packages/glmnet/index.html).

$\endgroup$
2
  • $\begingroup$ The only problem is, I have to be able to reproduce this with SPSS output. :( I'm still going to try it though! $\endgroup$ Nov 24, 2010 at 4:41
  • $\begingroup$ Tried it out, can't seem to get predict.glmnet() to work. Is there any magic that needs to be happening when you set newx? $\endgroup$ Nov 24, 2010 at 5:09
2
$\begingroup$

I know your question is about logistic regression and as it is a homework assignment so your approach may be constrained. However, if your interest is in interactions and accuracy of classification, it might be interesting to use something like CART to model this.

Here's some R code to do produce the basic tree. I've set rpart loose on the enire data frame here. Perhaps not the best approach without some prior knowledge and a cross validation method:

library(foreign)
df <- read.spss(file="http://www.bertelsen.ca/R/logistic-regression.sav", use.value.labels=T, to.data.frame=T) 
library(rpart) 
fit<-rpart(default~.,method="anova",data=df)
 pfit<- prune(fit, cp=   fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"])

# plot the pruned tree 
 plot(pfit, uniform=TRUE, 
   main="Pruned Classification Tree for Loan Default")
text(pfit, use.n=TRUE, all=TRUE, cex=.8)

I'm not sure right off how to produce the classifcation table. It shouldn't be too hard from the predicted values from the model object and the original values. Anyone have any tips here?

$\endgroup$
3
  • $\begingroup$ Classification and Regression Trees? That is actually a second part to the assignment. After I've maximized the classification I have to classify based on probability deciles. $\endgroup$ Nov 24, 2010 at 6:44
  • $\begingroup$ Actually, someone helped me with producing the classification table in this related question: stats.stackexchange.com/questions/4832/… Thanks for the example with R, much appreciated, I found similar instruction on the quick-r website. Although, for this puprpose I'm forced to apply CHAID in SPSS. $\endgroup$ Nov 24, 2010 at 7:00
  • $\begingroup$ predict method for prediction, table(originalClasses,predictedClasses) for table construction. I tried RF (usually it has accuracy as in overfitted CART, but no overfit) and the result was not much better than glm. $\endgroup$
    – user88
    Nov 24, 2010 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.