1
$\begingroup$

I'm analyzing a dataset to predict whether a customer defaults on a loan. The problem is, the dataset only contains observations on customers who were offered a loan and accepted (ie. there is no data on those who were refused a loan).

The problem here is that when applying the model to new data, there is an assumption that all of the customers are approved, which is not the case with the new data.

The sampling literature calls this "unit non-response" and methods for dealing with the problem are called "reject inference." While the literature suggests gathering more data, I'm wondering if it is possible adjust for unit non-response using stochastic Bayesian process.

The model works like this: there are three variables: c_default (whether the individual defaults), credit (credit score), and accept (whether the loan was accepted). The model is trying to predict the probability of default. It is a logistic regression run through Stan using the wrappers in the rethinking package.

The simulation of the data makes the assumption that those who are rejected (who we never see) have a credit score that is 30% lower. The criteria for rejection, regardless of group, is a credit score below 80.

library(rethinking)

approval <- c(rep(1, 50), rep(0, 50))
#data from accepted loans (we have this)
set.seed(1234); credit_accept <- rnorm(50, mean = 100, sd = 10); hist(credit_accept)
default_accept <- ifelse(credit_accept <80, 1, 0 )

#data from not accepted loans (not available, assume credit score is 30% lower)
set.seed(2345); credit_reject <- rnorm(50, mean = 70, sd = 10); hist(credit_reject)
default_reject <- ifelse(credit_reject < 80, 1, 0) #same rejection rate

credit <- c(credit_accept, credit_reject)
c_default <- c(default_accept, default_reject)
d <- cbind(approval, credit, c_default); d <- as.data.frame(d)
colnames(d) <- c("approval", "credit", "c_default")

The model is then trained on the simulated data with adaptive priors.

#logit
b_model <- map2stan(
  alist(
    c_default ~ dbinom( 1 , p ) ,
    logit(p) <- b_0 + b_1*credit + (b_2 + b_3*credit) * approval,
    ###prior distributions
    b_0 ~ dnorm( 0 , 2) ,
    b_1 ~ dnorm( b_1_mu, b_1_sig),
    b_1_mu ~ dnorm(0, 1),
    b_1_sig ~ dcauchy(0,1),
    b_2 ~ dnorm(b_2_mu, b_2_sig), 
    b_2_mu ~ dnorm(0,1),
    b_2_sig ~ dcauchy(0,1),
    b_3 ~ dnorm(b_3_mu, b_3_sig),
    b_3_mu ~ dnorm(0,1),
    b_3_sig ~ dcauchy(0,1)),
  data=d , chains=1, cores=4 , iter=1000 ) 

#precis
precis(b_model)

         mean     sd  5.5%  94.5% n_eff Rhat
b_0      4.69   1.67  2.04   7.35    93 1.00
b_1     -0.04   0.02 -0.08   0.00    83 1.00
b_1_mu  -0.05   0.56 -0.97   0.93   135 1.00
b_1_sig  0.82   1.01  0.05   2.43   131 1.00
b_2     70.84  67.58  0.04 190.15    67 1.02
b_2_mu   0.06   0.98 -1.45   1.68   306 1.00
b_2_sig 83.28 118.38  0.70 289.58   125 1.00
b_3     -0.90   0.83 -2.39  -0.04    67 1.02
b_3_mu  -0.50   0.75 -1.82   0.61   113 1.00
b_3_sig  1.24   2.37  0.04   3.88   234 1.00

We can see that approval is a crucial factor in the model.

At this point, we can take actual data from approved customers only and predict whether they default. This data is simulated, but all observations come from the "approved" sample with mean credit score of 100.

#test with new data, all approved
approval <- c(rep(1, 100))
set.seed(894); credit <- rnorm(100, mean = 100, sd = 10); hist(credit)
c_default <- ifelse(credit < 80, 1, 0 )
newdata <- cbind(approval, credit, c_default); newdata <- as.data.frame(newdata)
colnames(newdata) <- c("approval", "credit", "c_default")

#simulate
sim.m <- sim(b_model, data=newdata, n=500)
mean.prob <- apply(sim.m, 2, mean)

###
pred <- ifelse(mean.prob > .5, 1, 0)
truth <- newdata$c_default
table(pred, truth)

> table(pred, truth)
    truth
pred  0  1
   0 94  0
   1  0  6

The model nails prediction under these assumptions.

In short, the model is built under two assumptions: rejected applicants have a credit score that is 30% less than those accepted, and there is a default threshold of 80%. Thoughts?

$\endgroup$
  • $\begingroup$ Training a model on one population (those who receive a loan) and testing it on another population (those who applied for a loan) will produce unpredictable results. It is a major no-no. Your hope is that the model based on training data will generalize to a population that is substantially similar to the training population. Anything else is just . . . bad. "Will a standing jump exercise help NBA players dunk basketballs?" "Yes" "Then why won't it help 4-year-olds dunk basketballs?" "Unfair question!" Use the prior to reduce the test data to a set substantially similar to the training set. $\endgroup$ – Peter Leopold Jul 2 at 1:52
  • $\begingroup$ Right, so I'm wondering how to adjust my approach to this situation. Is adjusting a prior a trusted approach? If so, where is the literature? Or should I try something completely different? I'm looking for some general guidance. $\endgroup$ – Blake Shurtz Jul 2 at 3:32
  • $\begingroup$ A prior can be any honestly-arrived-at and/or generally accepted sensibility about the data. You have to say what it is, and your results are contingent on the validity of your prior, but after you've enunciated it clearly, you're cleared to perform any subsequent analysis. Ideally, all stake-holders will have a chance to concur with your prior. If they also agree w/your model, they are obligated to accept your results. How to pick select a prior that is fair, useful & keeps people engaged is an important & complicated topic. There are no general rules except to get concurrence in writing! $\endgroup$ – Peter Leopold Jul 2 at 4:06
  • $\begingroup$ In your case, you can say "any application with unknown outcome with a set of metrics/features that exceed a successful application on at least half [Can we get to YES on "half"?] of the available score metrics will be deemed a successful application for present purposes." Is that fair, useful, and keeps people engaged? Yes? Really? I need to hear you say it? Yes? A little louder. YES! OK, then we have our prior, so let's get back to the main question . . . $\endgroup$ – Peter Leopold Jul 2 at 4:11
  • $\begingroup$ Alternatively, if you don't care for the apparently subjective nature of this level of decision making, you can always go back out and get some decent data, or find a secondary feature in your current data set that correlates with and acts as a surrogate for acceptance/rejection and use that. But if you can't, well, this is the best we can do. If your task is mission critical, then go get the good data! $\endgroup$ – Peter Leopold Jul 2 at 4:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.