I have always dealt with Machine Learning problems involving classification tasks. I now have the following problem: obtain the purchasing probabilities of users given a certain dataset.
Reading the literature it seems to me that for obtaining well-calibrated probabilities one first has to train the model as a classification problem (with a binary score as precision, recall, f1 etc) and then calibrate the probabilities with isotonic regression or platt scaling. By well-calibrated probabilities I mean resembling the diagonal (perfect prediction) in a reliability plot (predicted probabilities by model vs the relative frequency of positive outcomes).
The following Figure is an example of a NOT well-calibrated problem using Random Forest (red->diagonal, green-> average fraction of purchasing events):
To tackle this problem is it enough to do the GridSearchCV (parameter search) with log-loss as a performance metric?.
Another performance metric could be calibration and discrimination as explained in Yates & Bates 1991 but is not implemented in scikit-learn. Maybe it is possible using the make_scorer function.