Probability Calibration of Statistical Models

Question

I am trying to better learn about the Probability Calibration of Statistical Models . For example, if a Random Forest model is trained on a binary supervised classification problem :

library(mlbench)
library(randomForest)
data(sonar)

rf <- randomForest(Class ~ ., data=Sonar, importance=TRUE,
                        proximity=TRUE)

prob = predict(rf,Sonar,type="prob")

pred = data.frame(cbind(prob , rf$predicted))

plot(pred$M, pred$V3)
plot(hist(pred$M))

As far as I understand, Probability Calibration would involve fitting a second statistical model (e.g. logistic regression) to the probability scores generated by the random forest model. For example:

pred$V3 = as.factor(ifelse(pred$V3 == "1", "0", "1"))
mylogit <- glm(V3 ~ M, data = pred, family = "binomial")

In a random forest model, if the generated probability is greater than 0.5, it belongs to the first class. If the probability is less than 0.5, it belongs to the other class. This is a very general rule which might not be very accurate. Probability Calibration might be able to solve this problem?

My Question: Can someone please tell me if my understanding of Probability Calibration is correct?

Thanks

Answer 1

Calibration is not related to going from a probability prediction to a hard classification. Even in a well-calibrated model, you might not want your threshold to be $0.5$. Imagine something like fraud detection. I don't want my bank alerting me willy-nilly, but if there's a $45\%$ chance that a transaction is fraudulent, that's worth my attention, even though it probably isn't fraud.

Calibration aims to investigate if the model is predicting the correct probabilities of class membership and correct the predictions if they are off. Sure, a model might come back with a probability prediction of $0.6$, but if almost all (say $95\%$) of the probability predictions of $0.6$ turn out to belong to class $1$, a probability of $0.6$ really means that there is a probability of $0.95$ of being in class $1$. We would say that the model is miscalibrated.

The magic of a function like rms::calibrate in R is that it can do calibration even when none of the probability predictions are the same. I confess that I have yet to figure out exactly what Harrell does in that function, however.

A potential resource is the sklearn documentation on model calibration, and I think this is a good statistics writeup, even for people who don't program in Python or do but not in sklearn.