I was having discussion with some colleagues and I would like to know some external opinion.
Description: We have to decide, for a given person, whether that person would choose item EXPENSIVE or item CHEAP. For item EXPENSIVE we have an unbalanced dataset: consisting of 6000 people, take up rate of 12% (because the item costs higher)... For item CHEAP we also have an unbalanced dataset: total size is 3000 people, take up rate is 18% (because the item costs less). Same variables for each model (slight different order in feature importance)
Problem: the challenge is to know if "person X" would take item EXPENSIVE instead of CHEAP
Approach: What my colleagues are proposing is to train "model A" only with EXPENSIVE data , and train "model B" only with CHEAP data... Then, they are evaluating "person X" in parallel on "model A" and "model B" and compare the output probability from each model. Each model fits 'OK' with a gini of approximately 0.7.
Question: My question would be:
- 1.- Is it safe to subtract probabilities from two different models when each just fits 'OK'?. I mean the lower the gini the higher the uncertainity of your prediction
- 2.- Can 2 different models be compared when each sample size is different and the take up rate is intrinsically different?
- 3.- How 'confident' would you be on results?
I think this is more an 'approach' question rather than 'hard numbers', I know it would be difficult to establish a comparison but I'm thinking (if valid) what could be the application for other problems.