0
$\begingroup$

I've created a hybrid model by taking an existing decision engine (TRUE/FALSE output) and supplementing it with a random forest classifier (TRUE/FALSE) model. The output of the hybrid model is produced by computing an OR from the predictions of the two models.

When evaluating the performance on historical data, I can produce a ROC for the random forest by computing the class probabilities based on the votes from all of the random trees combined. I cannot do this for the existing decision engine because it does not output probabilities. For the purpose of this analysis, I need to treat it as a black box.

How can I compute an ROC for the hybrid model? I would like to do this with R. I'm imagining that for each point on the ROC of the first model, I would need to compute the lift generated by adding the existing decision engine model.

$\endgroup$
  • $\begingroup$ I'm thinking that I can define the class probability of the hybrid model as 1 if the existing decision engine outputs TRUE, and equal to the random forest class probability if the existing decision engine outputs FALSE. Then I should see a lift in the ROC of the hybrid model as I've observed in the confusion matrix for the hybrid model compared to either model alone. $\endgroup$ – Bobby May 4 '18 at 21:54
0
$\begingroup$

A ROC models how you make decisions. They do assume a continous distribution (see https://www.ncbi.nlm.nih.gov/m/pubmed/11293781/ ) but if you are combining two binary models then the most states you will have are 4, 3 if the two input models are given the same weight. It would make more sense to me to use a contingency table.

$\endgroup$
  • $\begingroup$ I’m combining one model with a binary output with another that has a continuous output for probability. So not exactly two binary models. $\endgroup$ – Bobby May 5 '18 at 6:09
  • $\begingroup$ Ah, you describe combining them after reduction to a binary decision. Please update your question to make it clear what exactly you are doing $\endgroup$ – ReneBt May 5 '18 at 6:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.