1
$\begingroup$

I have built a random forest model on a dataset with a large class imbalance, I have attempted to maximize area under the curve when predicting on the test set. I wish to make a suggestion on when the model should be updated. Currently the best idea I have is to monitor ROC curves on a frequent basis and update when they fall below a certain threshold. Is there another method of calculating model decay indicating when Random Forest requires a full rebuild in R? Thank you in advance.

$\endgroup$
6
  • $\begingroup$ what is it that you are predicting? Is your target class somehow related to at least an ordinal value? $\endgroup$
    – dnbwise
    Mar 19 '16 at 1:22
  • $\begingroup$ Sorry for not making it clearer, I am predicting if someone will take up a policy (1) or not (0) using random forest. With there being more cases of those taking up the policy. $\endgroup$
    – Daniel
    Mar 19 '16 at 2:29
  • $\begingroup$ ok, is there something measurable post hoc that would be associated with someone taking up the policy? For instance, a person taking up the policy results in a dollar amount different from them not taking up the policy. $\endgroup$
    – dnbwise
    Mar 19 '16 at 2:41
  • $\begingroup$ If they take up the policy a premium would be received which is profit. While them not taking up the policy will result in the loss of their future business. Predicting someone as taking up the policy while they do not, type ii error is more costly to operations due to advertisement. The specific profits figures are not available though. $\endgroup$
    – Daniel
    Mar 19 '16 at 3:30
  • $\begingroup$ if you have historical profit figures, then you could project the percent increase in profits as compared to historical within some boundary. If the profits fall below lower bound, then recalibrate. How is the business going to assess the actual model performance? Are you just provided a binary indicator? $\endgroup$
    – dnbwise
    Mar 19 '16 at 5:06
0
$\begingroup$

Since there is a class imbalance and the only way of monitoring performance appears to be using the binary target, then I would recommend using precision and recall. The cases that the algorithm predicts as a target is more important than the cases that the algorithm doesn't predict as a target (i.e. precision).

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.