1
$\begingroup$

I am trying to create a logistic regression model to predict whether a customer given a loan will be a bad or a good customer: bad meaning missing a certain amount of payments and good meaning frequent enough and in time with payments. For the purpose of the model I have coded Bad as 1 and Good as 0 and tried different combinations with the variables.

One of the models I have built has an AIC of 5383.7 and Gini coefficient of 0.416733. This is the result after I play around with the threshold:

     FALSE TRUE
  0  3327  638
  1   165   95

So the model guessed that 165 customers would be good, but they are bad, but also put 638 good customers into the bad customers group.

The second model I built has an AIC of 5734.6 (350.9 higher), but its Gini is 0.4190394 and is slightly better at predicting the bad customers:

     FALSE TRUE
  0  3537  673
  1   177  105

[UPDATE] Okay. After checking a few things - It turns out that one of the variables has missing values and the model excludes the observations that have them by default. Hence the difference in observations in my models. I know about multiple imputation, but I don't really feel alright with it. My question is should I impute the missing data or should I exclude it from the data set so I can compare models with different number of variables?

$\endgroup$

bumped to the homepage by Community 8 hours ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

migrated from stackoverflow.com Jun 19 '16 at 22:19

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ 1. Welcome to SO! 2. This is a CrossValidated question, not a SO question. 3. Use your cross-validation classification error rates in conjunction with use-case specific qualitative choice of the appropriate accuracy measure to use from the dozen or so that you can get from the cross-validated confusion matrix. $\endgroup$ – Hack-R Jun 19 '16 at 20:17
  • 1
    $\begingroup$ Hi, it looks like you are using different data for both models (all the numbers in the second matrix are higher than the corresponding value in the first matrix). Why don't you run both models over the same data? $\endgroup$ – Matias Thayer Jun 19 '16 at 20:20
  • $\begingroup$ Your title seems misleading. You don't seem concerned with how to deal with data that is missing in regression. $\endgroup$ – Roland Jun 20 '16 at 15:01
  • $\begingroup$ @Roland That is what I am asking at the end of my post... Should I use multiple imputation or should I delete observations with missing data in the additional variables I use in the model. $\endgroup$ – Emil Filipov Jun 20 '16 at 17:39
0
$\begingroup$

First of all, in the model with 2 more variables you have less observations so check that (probably because of missing values).

I wouldn't use AIC criterion in this case since it's more practical to assess which model did a better performance in the classification. The statistics "sensitivity", "specificity", "false negative" and "false positive" are more useful in this scenario, and you should regard the one that fits your objective.

If that is to identify bad customers accurately, maximize specificity.

example with the classification tables you provided, SAS prints each one of these for different values of pi

$\endgroup$
  • $\begingroup$ Well. I just noticed that. I am using the same data set for both models. Which the same settings on the model with the same threshold on the prediction. Why does it generate a different contingency table? $\endgroup$ – Emil Filipov Jun 20 '16 at 5:02
0
$\begingroup$

Others have noted that there are differences between the data sets - and you certainly can't compare these metrics across different data sets successfully. The rest of the answer is around which metric should be preferred.

The AIC and Gini Coefficient are measuring quite different things, so you need to pause to consider what they are.

The AIC exists to check whether additional variables or different combinations of variables in some sense offer 'value for money' - they explicitly check the relative fit of the model penalising for extra variables added to achieve it. The intention is to find the optimum tradeoff between number of variables and fit of the model.

The Gini coefficient only checks one side of that balance - how well the model splits your data into the two categories.

One danger you face if you only consider the Gini coefficient is that the model may not generalise as well as you hoped when applied to new data. This is especially the case if you persist with 'statistically insignificant' variables (you didn't state your criterion, but my best guess is that it's something similar to p-value>0.05)

AIC is by no means perfect or the final answer, but some diagnostics around stability etc. are needed to have confidence that the model will perform on the next data set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.