Differential model. Random Forest I was having  discussion with some colleagues and I would like to know some external opinion.
Description:
We have to decide, for a given person, whether that person would choose item EXPENSIVE or item CHEAP.
For item EXPENSIVE we have an unbalanced dataset: consisting of 6000 people, take up rate of 12% (because the item costs higher)... For item CHEAP we also have an unbalanced dataset: total size is 3000 people, take up rate is 18% (because the item costs less). Same variables for each model (slight different order in feature importance)
Problem:
the challenge is to know if "person X" would take item EXPENSIVE instead of CHEAP
Approach:
What my colleagues are proposing is to train "model A" only with EXPENSIVE data , and train "model B" only with CHEAP data... Then, they are evaluating "person X" in parallel on "model A" and "model B" and compare the output probability from each model. Each model fits 'OK' with a gini of approximately 0.7.
Question:
My question would be:

*

*1.- Is it safe to subtract probabilities from two different models when each just fits 'OK'?. I mean the lower the gini the higher the uncertainity of your prediction

*2.- Can 2 different models be compared when each sample size is different and the take up rate is intrinsically different?

*3.- How 'confident' would you be on results?

I think this is more an 'approach' question rather than 'hard numbers', I know it would be difficult to establish a comparison but I'm thinking (if valid) what could be the application for other problems.
 A: 
Problem: the challenge is to know if "person X" would take item EXPENSIVE instead of CHEAP


*

*Going by the base rates you report in your question, Person X might not want to buy anything; probably this should be accounted for in the model. Have you considered aggregating the two datasets, fitting a model for a multinomial (or ordered) outcome (no buy, vs. cheap buy, vs. expensive buy)? Then you get predicted probabilities for each possible outcome.


*I'm assuming that the two datasets are samples from the same population. If not, it might not make a lot of sense to fit one model on the two datasets, or to compare the probabilities from two models fitted on the separate datasets anyway. Then you would best stick to evaluating the probability for a buy vs. not using only one of the models.


*Assuming the same population, the suggested model A / model B approach seems sensible, but the default RF classification approach might not be optimal:
For probability estimation, a regression RF will likely perform better than a classification RF. In a classification RF, classifications of the individual trees are averaged over to obtain a 'probability'. This 'probability' performs well for classification, but might not be a good estimator of the probability; they tend to be more extreme. With e.g., very noisy data and very low base rate of buys, all trees might predict 'no buy': the 'probability' predicted by the RF might be 0, while the proportion of buys in each of the terminal nodes in which the new observation fell were above 0 (but below .5).
In a regression RF (where one would model the binary outcome as a 0-1 variable), the predictions of each tree are a proportion (proportion of buys in the respective terminal node), and the final prediction of the RF is an average over that. That might make for a better estimator of probabilities. See for example:
Malley, J. D., Kruppa, J., Dasgupta, A., Malley, K. G., & Ziegler, A. (2012). Probability machines: consistent probability estimation using nonparametric learning machines. Methods of Information in Medicine, 51(1), 74. https://doi.org/10.3414/ME00-01-0052
Sage, A. J., Genschel, U., & Nettleton, D. (2020). Tree aggregation for random forest class probability estimation. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(2), 134-150. https://doi.org/10.1002/sam.11446
A: point 1 is extremely relevant and I think is the best approach. It could be modelled as 3 or 4 classes:

*

*3 classes: "YES" CHEAP; "YES" EXPENSIVE; "NO" (expensive and cheap)

*4 classes: "YES" CHEAP; "YES" EXPENSIVE; "NO" CHEAP; "NO" EXPENSIVE="MAYBE" CHEAP

After some reading I realised the approach taken is called "Incremental Response Modelling". Long story short, it works OK with very simple cases but it performs badly when fitting 2 parallel RF. As you mentioned, each RF is fitted independently and the features that increase differentiation (rather than response for each class) are not considered. I found a great paper that confirms the issues discussed
Radcliffe, N. and Patrick D. Surry. “Real-World Uplift Modelling with Significance-Based Uplift Trees.” (2012).
https://stochasticsolutions.com/pdf/sig-based-up-trees.pdf
A: 
1.- Is it safe to subtract probabilities from two different models when each just fits 'OK'?. I mean the lower the gini the higher the uncertainity of your prediction

No, it's not safe, but there's more to it than the models fitting just ok.
Assuming your models give well calibrated probabilities (which may not be the case, see Marjolein's answer), model A gives you the probability of some individual buying the cheap item instead of buying nothing, while model B gives you the probability of that individual buying the expensive item instead of buying nothing. The alternative is important, if provided with more alternatives, people can change their preferences in unpredictable ways. Maybe someone didn't buy the cheap item because he is not interested in the product, or maybe because he thought it deserved a bigger investment (and he would buy the expensive item instead).
Also, it has been shown that alternatives can even change people preferences in irrational ways (the presence of a more expensive item can demotivate people from buying the cheaper item, and the cheaper alternative most probably will make seem the more expensive item more desiderable).
In conclusion, a situation with three alternatives cannot be studied putting together two analysis with two alternatives. Joining those two models like your colleagues suggest may give a decent approximation, but if both models are just "ok", you can expect the resulting model to be quite less than just "ok".
