1
$\begingroup$

I'm currently working on the exercises of the book 'Applied Predictive Modeling' by Kuhn and Johnson (using R and caret) and am stuck at the issue of 'Calibrating Probabilities'.

Exercise 12.3 shows a problem linked to openly available data on churning:

library(C50)
data(churn)
table(churnTrain$Class

First of all I fit some linear classification models (the chapter is also about linear classification models so I did not go further than that for the exercises).

I found that LDA had the best fit with a sensitivity of 0.24 and specificity of 0.96. All of the five models had a very low sensitivity so this is actually the problem of this task I guess.

But then later I try calibration. By that I try to refit probabilities of Naive Bayes and of another model. When doing the calibration curve it looks like the Naive Bayes recalibration fits very well!

But then I try a confusion matrix between the estimated values through the Naive Bayes correction and the testing sample and see that it became worse: a sensitivity of 0.12.

And this is what I don't understand: it looks on the curve as if Naive Bayes fits very well to the line - but so why is the final prediction worse? Did I maybe do something completely wrong?

I am attaching here a reduced code of what I've done: Thanks, Nebi

library(C50)
library(caret)
data(churn)
str(churnTrain)
table(churnTrain$churn)

tr_y<-churnTrain$churn
te_y<-churnTest$churn

data_temp<- predict(dummyVars(~ ., data = churnTrain[,-length(churnTrain)]), newdata = churnTrain[,-length(churnTrain)])
test_temp<- predict(dummyVars(~ ., data = churnTest[,-length(churnTest)]), newdata = churnTest[,-length(churnTest)])



predictorInfo<-nearZeroVar(data_temp, saveMetrics = TRUE)
summary(predictorInfo)
vector<-!predictorInfo$nzv
data_temp<-data_temp[,vector]
test_temp<-test_temp[,vector]





correlations = cor(data_temp)
highCorrpp <- findCorrelation(correlations)

data_temp <- data_temp[, -highCorrpp]
test_temp <- test_temp[, -highCorrpp]



#no missing values
sum(!complete.cases(data_temp))



ctrl<- trainControl(method="cv",number=10,
                    summaryFunction=twoClassSummary,
                    classProbs=TRUE)

tr_x<-data_temp

te_x<-test_temp

#######I CUT OUT HERE THE OTHER MODELS

ldaMod<-train(x=tr_x,y=tr_y,
              method="lda",
              metric="ROC",
              preProcess=c("center","scale","BoxCox"),
              tuneLength = 15,
              trControl=ctrl)
ldaMod


ldaPred<-predict(ldaMod,te_x)
confusionMatrix(ldaPred,te_y)

ldaProb<-predict(ldaMod,te_x, type="prob")


glm.rocCurve = pROC::roc( response=te_y, predictor=ldaProb[,1] )
glm.rocCurve
plot(glm.rocCurve,legacy.axes=TRUE)

comb_te<-as.data.frame(te_x)
comb_te$y<-te_y#sic damit y bei beiden tr_y heisst

comb_te$ldaProb<-ldaProb[,1]
liftCurve<- lift(y~ldaProb,data=comb_te)
liftCurve

xyplot(liftCurve,
       auto.key=list(columns=2,
                     lines=TRUE,
                     points=FALSE))



calCurve <- calibration(y~ldaProb,data=comb_te)
calCurve
xyplot(calCurve,auto.key=list(columns=2))

comb_train<-as.data.frame(tr_x)
comb_train$y<-tr_y
sub_two<-predict(ldaMod,tr_x, type="prob")
comb_train$ldaProb<-sub_two[,1]



sigmoidalCal<-glm(relevel(y,ref="no")~ldaProb,
                  data=comb_train,
                  family=binomial)
coef(summary(sigmoidalCal))

sigmoidProbs<-predict(sigmoidalCal,newdata=comb_te[,"ldaProb",drop=FALSE],
                      type="response")


comb_te$LDAsigmoid<-sigmoidProbs

library(klaR)
BayesCal<-NaiveBayes(y~ldaProb,data=comb_train,usekernel=TRUE)
BayesProb<-predict(BayesCal,newdata=comb_te[,"ldaProb",drop=FALSE])
comb_te$LDABayes<-BayesProb$posterior[,"yes"]
head(comb_te)

#this is the curve that shows that NaiveBayes is performing well!
calCurve2<-calibration(y~ldaProb+LDAsigmoid+LDABayes, data=comb_te)
xyplot(calCurve2,auto.key=list(columns=3))

#and here is the confusion matrix showing that sensitivity is even lower!
confusionMatrix(BayesProb$class,comb_te$y)
$\endgroup$
1
  • $\begingroup$ I run your example code. Please note that all models have rather low Cohen's $k$ ($\leq 0.25$). While clearly no particular metric is a panacea, this suggests that none of them really does very good to begin with. $\endgroup$
    – usεr11852
    Commented Oct 8, 2017 at 13:46

1 Answer 1

2
$\begingroup$

Sensitivity, specificity, and a confusion matrix are at odds with optimal prediction. See here for more information. Calibration and predictive discrimination are all important. Calibration should be assessed using a flexible continuous calibration curve, accounting for overfitting. Predictive discrimination should be assessed using proper accuracy scoring rules combined with the semi-proper $c$-index (AUROC).

$\endgroup$
2
  • 1
    $\begingroup$ Thank you for your answer and insightful article. I understand that the AUROC is for optimal prediction overall more important than sensitivity and specificity. However, I am also in several aspects more confused now than before. I will try to find out more about these issues. For example in such an example, where 'yes' is so badly predicted, I thought that it would be worth to make a trade-off between ROC and sensitivity. Also the differentiation that you mention between the calibration on the one hand and the predictive discrimination makes me think... $\endgroup$ Commented Sep 17, 2017 at 14:35
  • 1
    $\begingroup$ That is exactly not the way to think about it in my humble opinion. Please read the article carefully at the link above and read also this. $\endgroup$ Commented Sep 17, 2017 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.