3
$\begingroup$

I have a logistic regression $\text{logit} ( p_i ) = \beta_0 + \sum_j \beta_j x_{ij}$ with a binary response variable that I'd like to form a kind of scorecard from. By creating a scorecard I simply mean taking logistic regression output and binning, cutting, or otherwise splitting it to determine (a) whether an action should be taken, and then further (b) at what scale the action should be taken (for example whether (a) a loan should be made and (b) what interest rate it should be at).

One way to handle (a) is to transform my output into a probability $p$ for each observation and then take $(1-p)$ multiplied by say 1000 to get a score $0 \leq s \leq 100$, from which (I think) I can take the K-S statistic as the cut-off.

I think this is probably a pretty weak method at least because I'm only getting one cut and not a set of bins above the cut-off with which to further rate observations.

With that in mind I have two questions:

1) How would you improve on the method above?

2) What are some ways to go about getting from the results of (a) to (b).

$\endgroup$
  • $\begingroup$ Logistic regressions tend to be well-calibrated. Have you checked your model's calibration? Also, could you expand/define "scorecard"? You could use the Logistic Regression result as a score [0-1] and multiply it by 100. But if "scorecard" involves more of a red/yellow/green or red/green cutoff, that's another matter. $\endgroup$ – Wayne Sep 15 '16 at 14:40
  • $\begingroup$ @Wayne Updated the question in an effort to be more specific about what I'm looking for. $\endgroup$ – 114 Sep 15 '16 at 15:24
  • $\begingroup$ Good, thanks. What's the target of the logistic regression? I'm assuming something like either "successful loan" or "bad loan", so you're wanting to split up the [0,1] logistic regression output into bands of something like "High Risk -- No Loan", "Medium Risk -- Higher interest rate loan", or "Low Risk -- Lower interest rate loan". Is that the target and your goal with the output? If your target is, say, "Bad Loan" and your regression is well-calibrated, you just need SME's to decide at what probability points (in [0,1]) they want to cut things off. $\endgroup$ – Wayne Sep 15 '16 at 15:35
  • $\begingroup$ That is, if your target is "Bad Loan" and the regression is well-calibrated, a score of 0.2 says there's approximately a 20% chance of this loan going Bad, while a 0.92 says there's approximately a 92% change of this loan going Bad. (In the opinion of your model, based on its training data.) Then it's just a matter of policy determining how much risk they're willing to tolerate and yet still extend a loan, or how low a risk they require before giving better terms. $\endgroup$ – Wayne Sep 15 '16 at 15:40
  • 1
    $\begingroup$ Then you can do an ''ordered logistic regression'', if will give you the coefficients $\beta$ but also the cut-points that you are looking for. I think in R it is with the function 'polr'. Your dependent variable is now the ''action class'', i.e. loan with low intrest, mocan with high intrest, so an ordered factor (in stead of 0/1 putcome) and you get coefficients and cut-points as result $\endgroup$ – user83346 Sep 15 '16 at 16:40
4
+50
$\begingroup$

You should try a technique called ''ordered logistic regression''; while logistic regression has a binary outcome as the dependent variable, the ''ordered logistic regression'' has an ordered categorical outcome as the dependent variable.

So if you have historical data on the $x_{ij}$ and historical data on the output class like ''no loan granted'', ''high interest loan granted'' 'low interest loan granted', ... (ordered) then you can apply ordered logistic regression.

The output of the ordered logistic regression are the estimates for the $\beta$'s but also the cut-points.

So using ordered logistic regression you will be able to compute a score (a probability or a real number (the log-odds of the probability)) and then it will also give you thresholds $T_1, T_2, ...$. These thresholds are estimated by the ordered logistic regression, so after estimating you know these.

Using the score and the thresholds you will be able to predict the outcome i.e. one of (see supra) ''no loan granted'', ''high interest loan granted'' 'low interest loan granted', ... (ordered).

E.g. It will tell you that if the score is between the thresholds $T_i=4$ and $T_{i+1}=9$ (the numbers are just examples of what the estimation may give you, so the 4 and 9 come out of the estimation) then it is ''high intrest loan granted'' (just an example of how you can predict).

In R it can be done using the polr function.

$\endgroup$
  • 1
    $\begingroup$ Nice approach! In R, the default would be polr in the package MASS, but you could also use a Bayesian approach with rstanarm's polr or use brm in package brms. (Both of these options use Stan under the hood.) $\endgroup$ – Wayne Sep 15 '16 at 17:19
1
$\begingroup$

So let's say that you have split your data into train (development) and test, chosen some optimum bins for your variables, and created your logistic regression model, and your target is "loan going bad" (credit to @Wayne).

The model will give you the probability of a loan going bad. You can of course multiply this by a 1000 to create more of "score feeling", which is also good when you're presenting to people not from a statistics background (it just makes it a bit more simpler).

Now when you wanna choose a cut off point (on the score) for taking action, this becomes a very subjective process, and I would not recommend taking the KS statistic as your cut-off.

What I usually do is that I split my data into equal tiles, 10 tiles most of the time, you can go to 20 as well if you wanna get a more refined look at your model.

By tiling, you will get your data split equally, and you will be able to plot your predicted probabilities against your observed probabilities which can also be used to check how good your model is. If there is a cash element (for e.g. if you looking at whether someone is going to pay back their loan, and you have a cash variable that tells you how much they have paid) you can also look at the cash distribution in the tiles.

I have already created a function that creates your tile table and outputs the graph which I will paste at the end of this answer. But the tiles can be used as a great guideline for choosing your cut-off points, because it creates a set of groups and you can choose which groups you would like to focus on. Obviously make sure to perform the same thing on your test data as well to make sure your results are robust!

If I have not been clear enough, do ask away! Hope this helped!

where model is the name of your model y is your target variable cash is your cash variable df is your dataset / dataframe mt and st are main title and sub title respectively

tile <- function(model, y = ds1$target, cash = NULL, df = ds1, mt = "Model Lift Chart", st = " ") {
  require(dplyr)

  p1 <- predict(model, df, type = "response")
  pred <- df$pred <- p1

  df$Tile <- ntile(pred , 10)

  if (is.null(cash)) {
  predict <- aggregate(pred, by=list(Tile = df$Tile), FUN=mean)
  observed <- aggregate(y, by=list(Tile = df$Tile), FUN=mean)
  count <- aggregate(pred, by=list(Tile = df$Tile),FUN=length)

  lift_mod <- data.frame(cbind(predict, observed, count))
  lift_mod$Tile.1 <- NULL
  lift_mod$Tile.2 <- NULL
  lift_mod$predict <- lift_mod$x
  lift_mod$observed <- lift_mod$x.1
  lift_mod$count <- lift_mod$x.2
  lift_mod$x <- NULL
  lift_mod$x.1 <- NULL
  lift_mod$x.2 <- NULL
  } else {
    predict <- aggregate(pred, by=list(Tile = df$Tile), FUN=mean)
    observed <- aggregate(y, by=list(Tile = df$Tile), FUN=mean)
    Cash_Mean <- aggregate(cash, by=list(Tile = df$Tile), FUN=mean) 
    count <- aggregate(pred, by=list(Tile = df$Tile),FUN=length)

    lift_mod <- data.frame(cbind(predict, observed, Cash_Mean, count))
    lift_mod$Tile.1 <- NULL
    lift_mod$Tile.2 <- NULL
    lift_mod$Tile.3 <- NULL
    lift_mod$predict <- lift_mod$x
    lift_mod$observed <- lift_mod$x.1
    lift_mod$Cash_Mean <- lift_mod$x.2
    lift_mod$count <- lift_mod$x.3
    lift_mod$x <- NULL
    lift_mod$x.1 <- NULL
    lift_mod$x.2 <- NULL
    lift_mod$x.3 <- NULL
  }
  print(lift_mod)
  a <- sum(lift_mod$Cash_Mean[1:3])/sum(lift_mod$Cash_Mean)
  b <- sum(lift_mod$Cash_Mean[8:10])/sum(lift_mod$Cash_Mean)
  print(sprintf("Bottom 3 = %.*f", 3, a))
  print(sprintf("Top 3 = %.*f", 3, b))
  mplot(lift_mod$Tile, lift_mod$predict, lift_mod$observed, mt=mt, st=st, xl = "Tile", yl = "Output")
  legend(x = "topleft", legend = c("Predict", "Actual"), lty = c(1, 1), col = c("steelblue2", "firebrick2"), pch = 19)
  t <- lift_mod$observed
  v <- t[2]
  w <- t[8]
  text(2, v, sprintf("Bottom 3 = %.*f%s", 1, a*100, "%"), col = "forestgreen", pos = 3, cex = 1.1)
  text(8, w, sprintf("Top 3 = %.*f%s", 1, b*100, "%"), col = "forestgreen", pos = 3, cex = 1.1)
  z <- lift_mod
}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.