0
$\begingroup$

I've used random forests for a while, in both regression and classification applications, and usually found that prediction quality does not increase significantly above 250 trees at the most. According to what I've read on CrossValidated or elsewhere, this is in agreement with the theory.

However, I'm currently working with a new dataset -- I can't say much about it except that it has roughly 200 columns, a few million rows and consists of what I'd call "business data", a mix of numbers (largely currency values) and one-hot encoded categorical data. We first tried 100-250 trees and got okay-ish results when predicting on our validation data. Someone in the team tried a much higher number of trees (800), and the results improved dramatically.

What could be the cause of this? We've experimented a bit with other metaparameters, but for the sake of this question let's assume that the others are as default in sklearn's RandomForestClassifier.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ The optimal number of trees is dependent on the properties of your dataset, so it is widely recommended that you try different numbers of trees. It's easy to examine how the error rate changes with tree number to help you choose. In the datasets I have worked with, 250 is quite low, and 2000-5000 ended up being the best choice, balancing error rate and computation time. $\endgroup$
    – mkt
    Jul 19, 2017 at 12:26

1 Answer 1

Reset to default
2
$\begingroup$

Random forest technically has only one hp: the # estimators E. By increasing E, we can reduce variance (a pure RF has no bounds on max_depth, e.g. (See Breiman's page). Especially with many predictors (or perhaps many spurious predictors) or some other characteristic of the data, E might need to be very large. Each tree grown to full size is an extremely overfit classifier; only as part of an ensemble does it give any kind of reasonable accuracy. I have seen it described elsewhere on this site that you must simply "add estimators until RF converges". The point of convergence depends on the dataset. It might be worth plotting learning curves to see whether the previous ensembles with fewer estimators were overfitting.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.