Is there a method to compare prediction from different models?

Question

I have a problem where I need to classify whether a client is likely to buy or not a service, I have around 10 service and 10.000 times more clients than service, and also rules that do not allow certain clients to buy specific service, say: a teenager can not buy alcohol delivery.

What I'm currently doing is to make a model for each of the 10 products, my problem is:

For a client, how can I compare the prediction of each of the models: say I model 1 return a probability of buying the service_1 of 0.4 and service_2 from model 2 has a probability of 0.6 can I directly compare those values? given that it is possible that the service1 is overall less likely to be bought

The history I have is the list of clients, some of the characteristics and whether they have or not a service.

Answer 1

can I directly compare those values?

Yes, provided all the models are well-calibrated (calibration) (or, perhaps, all badly-calibrated in the same way).

... given that it is possible that the service1 is overall less likely to be bought

This becomes very similar to the common question about class imbalance, where the general advice around here is to separate decision-making from modeling.

If the models are all well-calibrated, then your scenario puts a 40% chance of buying service_1 and a 60% chance of buying service_2, and that's that. If every customer comes out as more likely to purchase service_2 as a result of service_1 being less popular, then...well, that's just the truth of the matter. If you need to push service_1 despite that, then you can make different decisions based on the probabilities, suggesting service_1 even when it's less likely to be purchased, etc.

Answer 2

You can (and should) assess the quality of probabilistic predictions using a proper scoring rule. We have a scoring-rules tag, and there is a Wikipedia page. Note that most scoring rules are "negatively oriented", i.e., lower is better - but some people prefer the reverse orientation, by including (or removing) a minus sign.

Apply a scoring rule, e.g., the logatithmic score, to both models' predictions, and average them. The model with the lower (or higher, see above) scoring rule does a better job.