How accurate is evaluating classifiers' performance based on calibrated scores?

Question

Some classifiers do not give a probability estimate for whether or not a test point belongs to a class (i.e. P(c|x)). Instead, they provide scores which can then be calibrated into probabilities. Platt calibration is one method for doing so.

I need to calibrate the scores (of SVM classifiers) for two reasons:

First: as I am using k-fold CV, I would like to "sort the individual scores from all folds together into a single ROC curve and then compute the area of this curve". However, "it is not meaningful to compare the scores from different folds... unless the classifier is calibrated to output probabilities rather than just scores". These quotes are from THIS article in which the issue is explained in details.

Second: I would also like to use MSE instead of the classification accuracy as I assume the former will give more information about the performance of classifiers.

Since calibration is based on training another model (i.e. sigmoid function in Platt calibration), I am concerned about the influence of the accuracy of this additional model and its estimated probabilities on the evaluation of the classifiers' performance and, consequently, on the hyperparameters that get finally selected. Am I right to be concerned about that?

Answer 1

If you want to measure overall performance, I recommend approaches like nested cross-validation instead of comparing the output of models trained on different folds. The latter isn't the way to go.

I don't recommend Platt scaling in general. If you really need probabilities, don't use an SVM. You are almost always far better off with (kernel) logistic regression than with an SVM+Platt scaling.

You are right to distrust discrete metrics like accuracy that are based on a single (arbitrary) cutoff on the decision values of your model. If you have probabilistic output, you should consider using proper scoring rules such as Brier score, which is closely related to MSE, or log loss. Alternatively, since you don't really get reliable probabilistic output from an SVM, you can use metrics like area under the Receiver Operator Characteristic curve (aka concordance), though be aware that these are less sensitive than proper scoring rules.