Choosing a better model and dealing with missing data?

Question

I am trying to create a logistic regression model to predict whether a customer given a loan will be a bad or a good customer: bad meaning missing a certain amount of payments and good meaning frequent enough and in time with payments. For the purpose of the model I have coded Bad as 1 and Good as 0 and tried different combinations with the variables.

One of the models I have built has an AIC of 5383.7 and Gini coefficient of 0.416733. This is the result after I play around with the threshold:

     FALSE TRUE
  0  3327  638
  1   165   95

So the model guessed that 165 customers would be good, but they are bad, but also put 638 good customers into the bad customers group.

The second model I built has an AIC of 5734.6 (350.9 higher), but its Gini is 0.4190394 and is slightly better at predicting the bad customers:

     FALSE TRUE
  0  3537  673
  1   177  105

[UPDATE] Okay. After checking a few things - It turns out that one of the variables has missing values and the model excludes the observations that have them by default. Hence the difference in observations in my models. I know about multiple imputation, but I don't really feel alright with it. My question is should I impute the missing data or should I exclude it from the data set so I can compare models with different number of variables?

Answer 1

Others have noted that there are differences between the data sets - and you certainly can't compare these metrics across different data sets successfully. The rest of the answer is around which metric should be preferred.

The AIC and Gini Coefficient are measuring quite different things, so you need to pause to consider what they are.

The AIC exists to check whether additional variables or different combinations of variables in some sense offer 'value for money' - they explicitly check the relative fit of the model penalising for extra variables added to achieve it. The intention is to find the optimum tradeoff between number of variables and fit of the model.

The Gini coefficient only checks one side of that balance - how well the model splits your data into the two categories.

One danger you face if you only consider the Gini coefficient is that the model may not generalise as well as you hoped when applied to new data. This is especially the case if you persist with 'statistically insignificant' variables (you didn't state your criterion, but my best guess is that it's something similar to p-value>0.05)

AIC is by no means perfect or the final answer, but some diagnostics around stability etc. are needed to have confidence that the model will perform on the next data set.

Answer 2

First of all, in the model with 2 more variables you have less observations so check that (probably because of missing values).

I wouldn't use AIC criterion in this case since it's more practical to assess which model did a better performance in the classification. The statistics "sensitivity", "specificity", "false negative" and "false positive" are more useful in this scenario, and you should regard the one that fits your objective.

If that is to identify bad customers accurately, maximize specificity.