Comparability of classifier probabiliy estimates Consider that you have 3 classification models ($model_1$, $model_2$ and $model_3$) that are designed to model whether or not customers are interested in different products of your company (binary classification), e.g. $product_1$, $product_2$ and $product_3$.
While the customers are the same in all three cases, the data about them that goes into the different models might differ.
For a set of customers you now could have all three models compute the class probabilities of the positive class (customer is interested in product) for the customers.
Given that the three models use different algorithms, can the estimated probabilities ever be reliably compared to one another? For example could they be used in deciding which product to approach the customer about or is this problematic? If it is problematic, why is it?
Sidenote: why would I want to compare probabilities in the first place?
The estimated probability can factor into further computation in ways that a simple classification can not. Consider the case that products 1, 2 and 3 offer different monetary profits, while approaching the customer about a product (via letter, outbound service call or other channels) incurs costs. Using an estimated probability here allows for computing some form of expected gain of the different decisions, which a classification does not.
My first thought on this was that the differences in calibration of the estimated probabilities are the first problem to be addressed.   As [1] states:

"We show that maximum margin methods such as boosted trees and boosted
stumps push probability mass away from 0 and 1 yielding a
characteristic sigmoid shaped distortion in the predicted
probabilities. Models such as Naive Bayes, which make unrealistic
independence assumptions, push probabilities toward 0 and 1. Other
models such as neural nets and bagged trees do not have these biases
and predict well calibrated probabilities" .

Based on this my assumption is that some models are more likely to output estimated probabilities of for example 0.9, while the highest estimated probability of another model might end at 0.8 . In this case, the former model would always 'take the cake' in direct comparison.
Of course the models could undergo calibration, but would that make them comparable? I have reservations against simply comparing the model estimates like this even after calibration but I cannot put the finger on what exactly seems problematic about it to me.
[1] Niculescu-Mizil, A., & Caruana, R. (2005, August). Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning (pp. 625-632).
 A: Comparing the models and taking the highest probability is not a good idea as it doesn't correlate with accuracy . You should use one model which is the most accurate. This model can be one of the three or ensemble of models etc.
You can compare their accuracy by calculating offline measures e.g. AUC & log loss on a common validation set. i.e. you can have a validation set with ground truth labels, have the prediction of each model, and then you calculate the AUC & log loss for each model and compare the results. Another approach is to compare them using ranking measure like: NDCG
"could they be used in deciding which product to approach the customer" - Yes, if you have good offline measures for your model (e.g. high AUC) you can use it to recommend products. Though in machine learning it is never 100% guaranteed that good offline performance will lead to good online results (e.g. more money), it is usually correlated, and better models (better AUC) provide better results also online.
Regarding calibration - this is a different topic from accuracy. You can have more accurate model which is less calibrated. At least at the beginning you should focus on accuracy and ranking and only then, ask yourself if you need your model to be also calibrated. The difference is that accuracy is pointwise measure, and calibration care about the model average being closer to prediction average. If you also care about calibration you can have additional model like isotonic regression that will calibrate the model results.
