Some classifiers do not give a probability estimate for whether or not a test point belongs to a class (i.e. P(c|x)). Instead, they provide scores which can then be calibrated into probabilities. Platt calibration is one method for doing so.
I need to calibrate the scores (of SVM classifiers) for two reasons:
First: as I am using k-fold CV, I would like to "sort the individual scores from all folds together into a single ROC curve and then compute the area of this curve". However, "it is not meaningful to compare the scores from different folds... unless the classifier is calibrated to output probabilities rather than just scores". These quotes are from THIS article in which the issue is explained in details.
Second: I would also like to use MSE instead of the classification accuracy as I assume the former will give more information about the performance of classifiers.
Since calibration is based on training another model (i.e. sigmoid function in Platt calibration), I am concerned about the influence of the accuracy of this additional model and its estimated probabilities on the evaluation of the classifiers' performance and, consequently, on the hyperparameters that get finally selected. Am I right to be concerned about that?