I have a random forest model that attempts to classify a binary outcome (either "Good" or "Bad"). I am currently looking at ways in which I can evaluate the performance of the model when it is used on newer data to determine if the model is still performing acceptably.

The only performance metrics I can think off are:

  • Gini
  • Comparison of average actual vs predicted "bad rate"

Are there any other tests that are used to monitor how well a random forest model is performing?

Note: In another question ( Random Forest - Variable Importance over time ) I tried asking about how to explain the predictions of a random forest model. However, this question is not about model interpretability. My main goal here is to find a set of performance metrics that can help determine whether my model needs to be re-trained or not.

  • $\begingroup$ What's wrong with your garden-variety classification quality measures like accuracy, F1, AUROC, etc? $\endgroup$ – shadowtalker Aug 27 '16 at 12:57
  • $\begingroup$ I mentioned accuracy in my original post (actual vs predicted), likewise area under ROC curve is a linear transformation of gini. I have no problem in using the standard measures (forgot to add confusion matrices) - but I was wondering if there are any other non-standard techniques that others have found superior that I might be missing. $\endgroup$ – Jeebus Aug 27 '16 at 13:03

I'm not aware of any special performance measures for random forests, and I certainly see all manner of performance measures applied to them in the wild. I'm also not aware of any reason why random forests would need special treatment in this regard.

Generally, it's a good idea to use the objective function implied by your model as your performance metric. Bit random forests don't really have a unified objective function, being an ensemble of greedy decision trees. The underlying trees usually use Gini impurity or entropy as the splitting criterion, but these are not the same thing as their accuracy-measuring counterparts, so any similarity is purely spiritual.

So until someone smarter than me comes along with a different answer, my recommendation is to use what you're comfortable with and what makes sense for your problem.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.