1
$\begingroup$

I've been building a logistic regression model (using the "glm" method in caret). The training dataset is extremely imbalanced (99% of the observations in the majority class), so I've been trying to optimize the probability threshold during the resampling process using the train function from the caret package as described in this example of a svm model: Illustrative Example 5: Optimizing probability thresholds for class imbalances.

The idea is to get the classification parameters for different values of the probability thershold, like this:

threshold   ROC    Sens   Spec   Dist   ROC SD  Sens SD  Spec SD  Dist SD
 0.0100     0.957  1.000  0.000  1.000  0.0366  0.0000   0.0000   0.0000 
 0.0616     0.957  1.000  0.000  1.000  0.0366  0.0000   0.0000   0.0000 
 0.1132     0.957  1.000  0.000  1.000  0.0366  0.0000   0.0000   0.0000 
 0.1647     0.957  1.000  0.000  1.000  0.0366  0.0000   0.0000   0.0000 
  ...        ...                  ...                      ...      ...

I noticed that the 'glm' method in caret uses 0.5 as the probability cutoff value as can be seen in the predict function of the model:

code_glm <- getModelInfo("glm", regex = FALSE)[[1]]
code_glm$predict
function(modelFit, newdata, submodels = NULL) {
                if(!is.data.frame(newdata)) newdata <- as.data.frame(newdata)
                if(modelFit$problemType == "Classification") {
                  probs <-  predict(modelFit, newdata, type = "response")
                  out <- ifelse(probs < .5,
                                modelFit$obsLevel[1],
                                modelFit$obsLevel[2])
                } else {
                  out <- predict(modelFit, newdata, type = "response")
                }
                out
              }

Any ideas about how to pass a grid of probability cutoff values to the predict function shown above to get the optime cutoff value?

I've been trying to adapt the code from the example shown in the caret website, but I haven't been able to make it work. I think I'm finding difficult to understand how caret uses the model's interfaces...

$\endgroup$

2 Answers 2

4
$\begingroup$

You can vary the probability cutoff values over the range 0 to 1, and check the optimum cut off for maximum accuracy:

    logmodel <- glm(y~., data = data, family = binomial)

considering logmodel as your fitted model, which outputs the probabilities, use a function that calculates the accuracy of classification for each cut-off value like below

cutoffs <- seq(0.1,0.9,0.1)
accuracy <- NULL
for (i in seq(along = cutoffs)){
    prediction <- ifelse(logmodel$fitted.values >= cutoffs[i], 1, 0) #Predicting for cut-off
accuracy <- c(accuracy,length(which(data$y ==prediction))/length(prediction)*100)
}

And then you can visually explore the cutoff vs probability by plotting

plot(cutoffs, accuracy, pch =19,type='b',col= "steelblue",
     main ="Logistic Regression", xlab="Cutoff Level", ylab = "Accuracy %")

This will be the type of output:(I've added some ablines) enter image description here

$\endgroup$
3
  • $\begingroup$ Thanks a lot @Vikramnath. Just wondering from your answer... Should the accuracy vs cutoff analysis be performed using the training set or a different dataset? I was thinking of splitting my dataset in 3 different sets: training (60%), cutoff analysis (20%) and validation (20%) sets. I don't know if finding the optimal cutoff from the training or validations sets will lead me to overfitting... What do you think? $\endgroup$ Mar 5, 2016 at 19:40
  • $\begingroup$ I can't think of a hard and fast rule, but go for a k fold cross validation and apply the cutoff analysis. Select the then choose the probability value that showed the least variation during all validations. You can avoid any type of overfitting $\endgroup$ Mar 6, 2016 at 2:04
  • $\begingroup$ For instance, if one is predicting "bankruptcy", one can run a glm logit and calculate predicted probabilities of "bankruptcy, but since most people don't go bankrupt, it's zero-inflated; the default in caret is to classify those with a predicted probability > 0.5 as "likely to go bankrupt", but if the output of the accuracy thresholds you provided the code for above says the best threshold is 0.8, that means we should code as "likely to go bankrupt" those cases with a predicted probability > 0.8, right? $\endgroup$
    – coip
    Feb 15, 2018 at 23:38
0
$\begingroup$

There is also the threshold function from caret. Just like the answer by Vikram Venkat, this only works in trained models. So I think if you are cross-validating different versions of a model, you can only optimize the threshold for the final model, and you don't know if the optimal threshold will also be the optimal one generalizing to new data. So an adaptation of the mentioned example from the caret website would be preferable, but I am also unable to fully understand the working of the adaptation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.