# Calibrating Probabilities worse than Original Model even though better performance on calibration curve? (in R / Caret)

I'm currently working on the exercises of the book 'Applied Predictive Modeling' by Kuhn and Johnson (using R and caret) and am stuck at the issue of 'Calibrating Probabilities'.

Exercise 12.3 shows a problem linked to openly available data on churning:

library(C50)
data(churn)
table(churnTrain$Class  First of all I fit some linear classification models (the chapter is also about linear classification models so I did not go further than that for the exercises). I found that LDA had the best fit with a sensitivity of 0.24 and specificity of 0.96. All of the five models had a very low sensitivity so this is actually the problem of this task I guess. But then later I try calibration. By that I try to refit probabilities of Naive Bayes and of another model. When doing the calibration curve it looks like the Naive Bayes recalibration fits very well! But then I try a confusion matrix between the estimated values through the Naive Bayes correction and the testing sample and see that it became worse: a sensitivity of 0.12. And this is what I don't understand: it looks on the curve as if Naive Bayes fits very well to the line - but so why is the final prediction worse? Did I maybe do something completely wrong? I am attaching here a reduced code of what I've done: Thanks, Nebi library(C50) library(caret) data(churn) str(churnTrain) table(churnTrain$churn)

tr_y<-churnTrain$churn te_y<-churnTest$churn

data_temp<- predict(dummyVars(~ ., data = churnTrain[,-length(churnTrain)]), newdata = churnTrain[,-length(churnTrain)])
test_temp<- predict(dummyVars(~ ., data = churnTest[,-length(churnTest)]), newdata = churnTest[,-length(churnTest)])

predictorInfo<-nearZeroVar(data_temp, saveMetrics = TRUE)
summary(predictorInfo)
vector<-!predictorInfo$nzv data_temp<-data_temp[,vector] test_temp<-test_temp[,vector] correlations = cor(data_temp) highCorrpp <- findCorrelation(correlations) data_temp <- data_temp[, -highCorrpp] test_temp <- test_temp[, -highCorrpp] #no missing values sum(!complete.cases(data_temp)) ctrl<- trainControl(method="cv",number=10, summaryFunction=twoClassSummary, classProbs=TRUE) tr_x<-data_temp te_x<-test_temp #######I CUT OUT HERE THE OTHER MODELS ldaMod<-train(x=tr_x,y=tr_y, method="lda", metric="ROC", preProcess=c("center","scale","BoxCox"), tuneLength = 15, trControl=ctrl) ldaMod ldaPred<-predict(ldaMod,te_x) confusionMatrix(ldaPred,te_y) ldaProb<-predict(ldaMod,te_x, type="prob") glm.rocCurve = pROC::roc( response=te_y, predictor=ldaProb[,1] ) glm.rocCurve plot(glm.rocCurve,legacy.axes=TRUE) comb_te<-as.data.frame(te_x) comb_te$y<-te_y#sic damit y bei beiden tr_y heisst

comb_te$ldaProb<-ldaProb[,1] liftCurve<- lift(y~ldaProb,data=comb_te) liftCurve xyplot(liftCurve, auto.key=list(columns=2, lines=TRUE, points=FALSE)) calCurve <- calibration(y~ldaProb,data=comb_te) calCurve xyplot(calCurve,auto.key=list(columns=2)) comb_train<-as.data.frame(tr_x) comb_train$y<-tr_y
sub_two<-predict(ldaMod,tr_x, type="prob")
comb_train$ldaProb<-sub_two[,1] sigmoidalCal<-glm(relevel(y,ref="no")~ldaProb, data=comb_train, family=binomial) coef(summary(sigmoidalCal)) sigmoidProbs<-predict(sigmoidalCal,newdata=comb_te[,"ldaProb",drop=FALSE], type="response") comb_te$LDAsigmoid<-sigmoidProbs

library(klaR)
BayesCal<-NaiveBayes(y~ldaProb,data=comb_train,usekernel=TRUE)
BayesProb<-predict(BayesCal,newdata=comb_te[,"ldaProb",drop=FALSE])
comb_te$LDABayes<-BayesProb$posterior[,"yes"]

#this is the curve that shows that NaiveBayes is performing well!
calCurve2<-calibration(y~ldaProb+LDAsigmoid+LDABayes, data=comb_te)
xyplot(calCurve2,auto.key=list(columns=3))

#and here is the confusion matrix showing that sensitivity is even lower!
confusionMatrix(BayesProb$class,comb_te$y)

• I run your example code. Please note that all models have rather low Cohen's $k$ ($\leq 0.25$). While clearly no particular metric is a panacea, this suggests that none of them really does very good to begin with. Commented Oct 8, 2017 at 13:46

Sensitivity, specificity, and a confusion matrix are at odds with optimal prediction. See here for more information. Calibration and predictive discrimination are all important. Calibration should be assessed using a flexible continuous calibration curve, accounting for overfitting. Predictive discrimination should be assessed using proper accuracy scoring rules combined with the semi-proper $c$-index (AUROC).