0
$\begingroup$

I have a random forest model that attempts to classify a binary outcome (either "Good" or "Bad"). I am currently looking at ways in which I can evaluate the performance of the model when it is used on newer data to determine if the model is still performing acceptably.

The only performance metrics I can think off are:

  • Gini
  • Comparison of average actual vs predicted "bad rate"

Are there any other tests that are used to monitor how well a random forest model is performing?

Note: In another question ( Random Forest - Variable Importance over time ) I tried asking about how to explain the predictions of a random forest model. However, this question is not about model interpretability. My main goal here is to find a set of performance metrics that can help determine whether my model needs to be re-trained or not.

Improve this question
$\endgroup$
4
  • $\begingroup$ What's wrong with your garden-variety classification quality measures like accuracy, F1, AUROC, etc? $\endgroup$
    – shadowtalker
    Aug 27, 2016 at 12:57
  • $\begingroup$ I mentioned accuracy in my original post (actual vs predicted), likewise area under ROC curve is a linear transformation of gini. I have no problem in using the standard measures (forgot to add confusion matrices) - but I was wondering if there are any other non-standard techniques that others have found superior that I might be missing. $\endgroup$
    – Jeebus
    Aug 27, 2016 at 13:03
  • $\begingroup$ Area under the ROC curve is NOT a linear transformation of Gini. Gini index measures the within-node variance, ROC measures something different. I don't think you can easily compute Gini index on test observations, though. Mean squared error loss on predicted probabilities seems like a good choice then. $\endgroup$
    – Marjolein Fokkema
    Feb 25, 2021 at 12:42
  • $\begingroup$ I think the confusion here is that I was referring to the Gini coefficient whereas your comment is in reference to the Gini measure of impurity that is used in building decision tree nodes. en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity The gini coefficient is commonly used to measure the rank-ordering performance of a model and I believe it is linearly related to the area under the ROC curve. $\endgroup$
    – Jeebus
    Feb 26, 2021 at 13:24

1 Answer 1

Reset to default
1
$\begingroup$

I'm not aware of any special performance measures for random forests, and I certainly see all manner of performance measures applied to them in the wild. I'm also not aware of any reason why random forests would need special treatment in this regard.

Generally, it's a good idea to use the objective function implied by your model as your performance metric. Bit random forests don't really have a unified objective function, being an ensemble of greedy decision trees. The underlying trees usually use Gini impurity or entropy as the splitting criterion, but these are not the same thing as their accuracy-measuring counterparts, so any similarity is purely spiritual.

So until someone smarter than me comes along with a different answer, my recommendation is to use what you're comfortable with and what makes sense for your problem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.