Better Classification of default in logistic regression Full Disclosure: This is homework. I've included a link to the dataset ( http://www.bertelsen.ca/R/logistic-regression.sav )
My goal is to maximize the prediction of loan defaulters in this data set.  
Every model that I have come up with so far, predicts >90% of non-defaulters, but <40% of defaulters making the classification efficiency overall ~80%. So, I wonder if there are interaction effects between the variables? Within a logistic regression, other than testing each possible combination is there a way to identify potential interaction effects? Or alternatively a way to boost the efficiency of classification of defaulters. 
I'm stuck, any recommendations would be helpful in your choice of words, R-code or SPSS syntax. 
My primary variables are outlined in the following histogram and scatterplot (with the exception of the dichotomous variable)
A description of the primary variables: 
age: Age in years
employ: Years with current employer
address: Years at current address
income: Household income in thousands
debtinc: Debt to income ratio (x100)
creddebt: Credit card debt in thousands
othdebt: Other debt in thousands
default: Previously defaulted (dichotomous, yes/no, 0/1)
ed: Level of education (No HS, HS, Some College, College, Post-grad)

Additional variables are just transformations of the above. I also tried converting a few of the continuous variables into categorical variables and implementing them in the model, no luck there. 
If you'd like to pop it into R, quickly, here it is: 
## R Code
df <- read.spss(file="http://www.bertelsen.ca/R/logistic-regression.sav", use.value.labels=T, to.data.frame=T)



 A: In unbalanced datasets such as this, you can usually improve classification performance by moving away from using a fitted probability of .5 as your cutpoint for classifying cases into defaulters and non-defaulters. For example, I get correct classification rates of .88 and .58 with a cutpoint of .4 for a glm with all 2nd-order interactions. (Which probably leads to overfitting and seems to have some rank issues, but that's another story.)
Code:
m <- glm(default ~ (age + employ + address + income + debtinc + 
                    creddebt + othdebt + ed)^2,
   family=binomial(), data=df)
p <- predict(m, newdata=df, type="response")

getMisclass <- function(cutoff, p, labels){
   pred <- factor(1*(p > cutoff), labels=c("No Default", "Default")) 
   t <- table(pred, labels)
   cat("cutoff ", cutoff, ":\n")
   print(t)
   cat("correct    :", round(sum(t[c(1,4)])/sum(t), 2),"\n")
   cat("correct No :", round(t[1]/sum(t[,1]), 2),"\n")
   cat("correct Yes:", round(t[4]/sum(t[,2]), 2),"\n\n")
   invisible(t)
}
cutoffs <- seq(.1,.9,by=.1)
sapply(cutoffs, getMisclass, p=p, labels=df$default)

partial output:
cutoff  0.3 :
            labels
pred           No  Yes
  No Default 3004  352
  Default     740  903
correct    : 0.78 
correct No : 0.8 
correct Yes: 0.72 

cutoff  0.4 :
            labels
pred           No  Yes
  No Default 3278  532
  Default     466  723
correct    : 0.8 
correct No : 0.88 
correct Yes: 0.58 

cutoff  0.5 :
        labels
pred           No  Yes
  No Default 3493  685
  Default     251  570
correct    : 0.81 
correct No : 0.93 
correct Yes: 0.45 

cutoff  0.6 :
            labels
pred           No  Yes
  No Default 3606  824
  Default     138  431
correct    : 0.81 
correct No : 0.96 
correct Yes: 0.34 

A: In logistic regression, highly skewed distributions of outcome variables (where there are far more non-events to events or vis versa), the cut point or probability trigger does need to be adjusted, but it will not have much of an effect on overall classification efficieny.  This will always remain roughly the same, but you are currently under-classifying events since the "chance" probability in such a data set will always make you more likely to classify into non-events.  This needs to be adjusted for.  In fact, in such a situation it's not uncommon to see the overall efficiency of classification go down,since it was previously inflated by misscalculation due to chance.  
Think of it this way, if you have an event where 90% don't do it and 10% do it, then if you put everyone into the "don't do it" group, you automatically get 90% right, and that was without even trying, just pure chance, inflated by the skewness of it's distribution.  
The issue of interactions is unrelated to this skewing, and should be driven by theory.  You will most likely always improve classification by adding additional terms, including simply adding interactions, but you do so by often overfitting the model. You then have to go back and be able to interpret this.  
Matt P
Data Analyst, University of Illinois Urbana Champaign
A: I'm not a logistic regression expert, but isn't it just a problem of unbalanced data? Probably you have much more non-defaulters than defaulters what may shift the prediction to deal better with larger class. Try to kick out some non-defaulters and see what happens.
A: You might just try including all of the interaction effects. You can then use L1/L2-regularized logistic regression to minimize over-fitting and take advantage of any helpful features. I really like Hastie/Tibshirani's glmnet package (http://cran.r-project.org/web/packages/glmnet/index.html).
A: I know your question is about logistic regression and as it is a homework assignment so your approach may be constrained.  However, if your interest is in interactions and accuracy of classification, it might be interesting to use something like CART to model this.
Here's some R code to do produce the basic tree.  I've set rpart loose on the enire data frame here.  Perhaps not the best approach without some prior knowledge and a cross validation method:
library(foreign)
df <- read.spss(file="http://www.bertelsen.ca/R/logistic-regression.sav", use.value.labels=T, to.data.frame=T) 
library(rpart) 
fit<-rpart(default~.,method="anova",data=df)
 pfit<- prune(fit, cp=   fit$cptable[which.min(fit$cptable[,"xerror"]),"CP"])

# plot the pruned tree 
 plot(pfit, uniform=TRUE, 
   main="Pruned Classification Tree for Loan Default")
text(pfit, use.n=TRUE, all=TRUE, cex=.8)

I'm not sure right off how to produce the classifcation table.  It shouldn't be too hard from the predicted values from the model object and the original values.  Anyone have any tips here?
