1
$\begingroup$

I have a problem where I need to classify whether a client is likely to buy or not a service, I have around 10 service and 10.000 times more clients than service, and also rules that do not allow certain clients to buy specific service, say: a teenager can not buy alcohol delivery.

What I'm currently doing is to make a model for each of the 10 products, my problem is:

For a client, how can I compare the prediction of each of the models: say I model 1 return a probability of buying the service_1 of 0.4 and service_2 from model 2 has a probability of 0.6 can I directly compare those values? given that it is possible that the service1 is overall less likely to be bought

The history I have is the list of clients, some of the characteristics and whether they have or not a service.

$\endgroup$
2
  • $\begingroup$ Do you have the actual outcome (whether a given client actually did buy the product or not)? $\endgroup$ Mar 18 '20 at 16:05
  • $\begingroup$ @StephanKolassa Yes, I do. I have updated the question. $\endgroup$ Mar 18 '20 at 17:13
1
$\begingroup$

can I directly compare those values?

Yes, provided all the models are well-calibrated () (or, perhaps, all badly-calibrated in the same way).

... given that it is possible that the service1 is overall less likely to be bought

This becomes very similar to the common question about class imbalance, where the general advice around here is to separate decision-making from modeling.

If the models are all well-calibrated, then your scenario puts a 40% chance of buying service_1 and a 60% chance of buying service_2, and that's that. If every customer comes out as more likely to purchase service_2 as a result of service_1 being less popular, then...well, that's just the truth of the matter. If you need to push service_1 despite that, then you can make different decisions based on the probabilities, suggesting service_1 even when it's less likely to be purchased, etc.

$\endgroup$
0
$\begingroup$

You can (and should) assess the quality of probabilistic predictions using a proper scoring rule. We have a tag, and there is a Wikipedia page. Note that most scoring rules are "negatively oriented", i.e., lower is better - but some people prefer the reverse orientation, by including (or removing) a minus sign.

Apply a scoring rule, e.g., the logatithmic score, to both models' predictions, and average them. The model with the lower (or higher, see above) scoring rule does a better job.

$\endgroup$
2
  • 1
    $\begingroup$ The question is about models with different targets; it's not a question about which model is better. $\endgroup$ Mar 18 '20 at 20:25
  • $\begingroup$ @BenReiniger: ah. That may be. OP: can you confirm I misunderstood your question? $\endgroup$ Mar 19 '20 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.