Is there a method to compare prediction from different models?

I have a problem where I need to classify whether a client is likely to buy or not a service, I have around 10 service and 10.000 times more clients than service, and also rules that do not allow certain clients to buy specific service, say: a teenager can not buy alcohol delivery.

What I'm currently doing is to make a model for each of the 10 products, my problem is:

For a client, how can I compare the prediction of each of the models: say I model 1 return a probability of buying the service_1 of 0.4 and service_2 from model 2 has a probability of 0.6 can I directly compare those values? given that it is possible that the service1 is overall less likely to be bought

The history I have is the list of clients, some of the characteristics and whether they have or not a service.

• Do you have the actual outcome (whether a given client actually did buy the product or not)? Mar 18, 2020 at 16:05
• @StephanKolassa Yes, I do. I have updated the question. Mar 18, 2020 at 17:13

2 Answers

can I directly compare those values?

Yes, provided all the models are well-calibrated () (or, perhaps, all badly-calibrated in the same way).

... given that it is possible that the service1 is overall less likely to be bought

This becomes very similar to the common question about class imbalance, where the general advice around here is to separate decision-making from modeling.

If the models are all well-calibrated, then your scenario puts a 40% chance of buying service_1 and a 60% chance of buying service_2, and that's that. If every customer comes out as more likely to purchase service_2 as a result of service_1 being less popular, then...well, that's just the truth of the matter. If you need to push service_1 despite that, then you can make different decisions based on the probabilities, suggesting service_1 even when it's less likely to be purchased, etc.

You can (and should) assess the quality of probabilistic predictions using a proper scoring rule. We have a tag, and there is a Wikipedia page. Note that most scoring rules are "negatively oriented", i.e., lower is better - but some people prefer the reverse orientation, by including (or removing) a minus sign.

Apply a scoring rule, e.g., the logatithmic score, to both models' predictions, and average them. The model with the lower (or higher, see above) scoring rule does a better job.

• The question is about models with different targets; it's not a question about which model is better. Mar 18, 2020 at 20:25
• @BenReiniger: ah. That may be. OP: can you confirm I misunderstood your question? Mar 19, 2020 at 7:09