I am given a classifier for some disease that takes as input patient characteristics and has some sensitivity and specificity.
Hence the classifier is a function c(patient characteristics) = 1 or 0
I can then use Bayes rule to convert:
P(disease | c(patient characteristics) = 1) = P(c(patient characteristics) =1 |dz) P(dz) / P(c(patient characteristics) = 1)
Using classifier sensitivity and specificity to write P(c(patient characteristics) = 1 |dz)/P(c(patient characteristics) = 1).
So, even if all I have is a classifier (which made a decision), I can get some probability estimate.
The better approach is to develop an estimator directly for
P(disease | patient characteristics)
Eg using logistic regression or just never binarizing classifier output in the first place.
Classifiers are heavily criticized in medicine, and I agree that it's a poor choice to make a decision without patient and physician utilities, but why can't we just use the first method to convert the classifiers to probabilities?