1
$\begingroup$

I was having discussion with some colleagues and I would like to know some external opinion.

Description: We have to decide, for a given person, whether that person would choose item EXPENSIVE or item CHEAP. For item EXPENSIVE we have an unbalanced dataset: consisting of 6000 people, take up rate of 12% (because the item costs higher)... For item CHEAP we also have an unbalanced dataset: total size is 3000 people, take up rate is 18% (because the item costs less). Same variables for each model (slight different order in feature importance)

Problem: the challenge is to know if "person X" would take item EXPENSIVE instead of CHEAP

Approach: What my colleagues are proposing is to train "model A" only with EXPENSIVE data , and train "model B" only with CHEAP data... Then, they are evaluating "person X" in parallel on "model A" and "model B" and compare the output probability from each model. Each model fits 'OK' with a gini of approximately 0.7.

Question: My question would be:

  • 1.- Is it safe to subtract probabilities from two different models when each just fits 'OK'?. I mean the lower the gini the higher the uncertainity of your prediction
  • 2.- Can 2 different models be compared when each sample size is different and the take up rate is intrinsically different?
  • 3.- How 'confident' would you be on results?

I think this is more an 'approach' question rather than 'hard numbers', I know it would be difficult to establish a comparison but I'm thinking (if valid) what could be the application for other problems.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ 1) Think about log-odds it is seemingly awkward, but can be very useful. This means you divide probabilities then take the log, or that you subtract the log of probabilities from each other. 2) This is a great (!!) question because you can't compare apples and squid, they aren't both fruit. You can compare two models, one trained on "a" and one on "b" but you need to be aware of impacts to uncertainty of parameters, so uncertainty of result. 3) I would bootstrap the daylights out of it to make a CI, and then CV the CI to make sure I trust it. test test test. $\endgroup$
    – EngrStudent
    Commented Mar 2, 2021 at 16:46

3 Answers 3

Reset to default
2
$\begingroup$

Problem: the challenge is to know if "person X" would take item EXPENSIVE instead of CHEAP

  1. Going by the base rates you report in your question, Person X might not want to buy anything; probably this should be accounted for in the model. Have you considered aggregating the two datasets, fitting a model for a multinomial (or ordered) outcome (no buy, vs. cheap buy, vs. expensive buy)? Then you get predicted probabilities for each possible outcome.

  2. I'm assuming that the two datasets are samples from the same population. If not, it might not make a lot of sense to fit one model on the two datasets, or to compare the probabilities from two models fitted on the separate datasets anyway. Then you would best stick to evaluating the probability for a buy vs. not using only one of the models.

  3. Assuming the same population, the suggested model A / model B approach seems sensible, but the default RF classification approach might not be optimal:

For probability estimation, a regression RF will likely perform better than a classification RF. In a classification RF, classifications of the individual trees are averaged over to obtain a 'probability'. This 'probability' performs well for classification, but might not be a good estimator of the probability; they tend to be more extreme. With e.g., very noisy data and very low base rate of buys, all trees might predict 'no buy': the 'probability' predicted by the RF might be 0, while the proportion of buys in each of the terminal nodes in which the new observation fell were above 0 (but below .5).

In a regression RF (where one would model the binary outcome as a 0-1 variable), the predictions of each tree are a proportion (proportion of buys in the respective terminal node), and the final prediction of the RF is an average over that. That might make for a better estimator of probabilities. See for example:

Malley, J. D., Kruppa, J., Dasgupta, A., Malley, K. G., & Ziegler, A. (2012). Probability machines: consistent probability estimation using nonparametric learning machines. Methods of Information in Medicine, 51(1), 74. https://doi.org/10.3414/ME00-01-0052

Sage, A. J., Genschel, U., & Nettleton, D. (2020). Tree aggregation for random forest class probability estimation. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(2), 134-150. https://doi.org/10.1002/sam.11446

Improve this answer
$\endgroup$
2
  • $\begingroup$ Would'nt a SMOTE technique cover the "no-buy" predictionif we oversample the cheap and expensive buy classes? or is this problematic since both expensive and cheap are minority classes and this doesnt work so well at all? $\endgroup$
    – Patrick Bormann
    Commented Mar 6, 2021 at 14:08
  • $\begingroup$ I am not too familiar with SMOTE. I believe it works well for classification, not sure about predicted probabilities (e.g., oversampling minority class might make predicted probabilities of being in minority class unrealistically high). According to Frank Harrell, SMOTE would be needed only because of the use of an improper discontinuous accuracy scoring rule (twitter.com/f2harrell/status/1062424969366462473); seems a very sensible point to be. The regression RF suggested above minimizes the Brier score, yielding a continuous scoring rule, which might obviate the need for SMOTE. $\endgroup$
    – Marjolein Fokkema
    Commented Mar 7, 2021 at 0:12
1
$\begingroup$

point 1 is extremely relevant and I think is the best approach. It could be modelled as 3 or 4 classes:

  • 3 classes: "YES" CHEAP; "YES" EXPENSIVE; "NO" (expensive and cheap)
  • 4 classes: "YES" CHEAP; "YES" EXPENSIVE; "NO" CHEAP; "NO" EXPENSIVE="MAYBE" CHEAP

After some reading I realised the approach taken is called "Incremental Response Modelling". Long story short, it works OK with very simple cases but it performs badly when fitting 2 parallel RF. As you mentioned, each RF is fitted independently and the features that increase differentiation (rather than response for each class) are not considered. I found a great paper that confirms the issues discussed

Radcliffe, N. and Patrick D. Surry. “Real-World Uplift Modelling with Significance-Based Uplift Trees.” (2012).

https://stochasticsolutions.com/pdf/sig-based-up-trees.pdf

Improve this answer
$\endgroup$
1
  • $\begingroup$ Incremental response modeling allows to measure the incremental impact of a certain “treatment". The question does not refer to any such treatment. You refer to a paper from 2011, which reads "We include details of the Significance-Based Uplift Trees that have formed the core of the only packaged uplift modelling software currently available." Model-based recursive partitioning, implemented in R package party (and later partykit) has been able to do this since 2005, so I'm not sure it should be taken as an authoritative reference. $\endgroup$
    – Marjolein Fokkema
    Commented Mar 28, 2021 at 15:32
0
$\begingroup$

1.- Is it safe to subtract probabilities from two different models when each just fits 'OK'?. I mean the lower the gini the higher the uncertainity of your prediction

No, it's not safe, but there's more to it than the models fitting just ok.

Assuming your models give well calibrated probabilities (which may not be the case, see Marjolein's answer), model A gives you the probability of some individual buying the cheap item instead of buying nothing, while model B gives you the probability of that individual buying the expensive item instead of buying nothing. The alternative is important, if provided with more alternatives, people can change their preferences in unpredictable ways. Maybe someone didn't buy the cheap item because he is not interested in the product, or maybe because he thought it deserved a bigger investment (and he would buy the expensive item instead).

Also, it has been shown that alternatives can even change people preferences in irrational ways (the presence of a more expensive item can demotivate people from buying the cheaper item, and the cheaper alternative most probably will make seem the more expensive item more desiderable).

In conclusion, a situation with three alternatives cannot be studied putting together two analysis with two alternatives. Joining those two models like your colleagues suggest may give a decent approximation, but if both models are just "ok", you can expect the resulting model to be quite less than just "ok".

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.