Random forest classification improves dramatically when using a very high number of trees

Question

I've used random forests for a while, in both regression and classification applications, and usually found that prediction quality does not increase significantly above 250 trees at the most. According to what I've read on CrossValidated or elsewhere, this is in agreement with the theory.

However, I'm currently working with a new dataset -- I can't say much about it except that it has roughly 200 columns, a few million rows and consists of what I'd call "business data", a mix of numbers (largely currency values) and one-hot encoded categorical data. We first tried 100-250 trees and got okay-ish results when predicting on our validation data. Someone in the team tried a much higher number of trees (800), and the results improved dramatically.

What could be the cause of this? We've experimented a bit with other metaparameters, but for the sake of this question let's assume that the others are as default in sklearn's RandomForestClassifier.

Answer 1

Random forest technically has only one hp: the # estimators E. By increasing E, we can reduce variance (a pure RF has no bounds on max_depth, e.g. (See Breiman's page). Especially with many predictors (or perhaps many spurious predictors) or some other characteristic of the data, E might need to be very large. Each tree grown to full size is an extremely overfit classifier; only as part of an ensemble does it give any kind of reasonable accuracy. I have seen it described elsewhere on this site that you must simply "add estimators until RF converges". The point of convergence depends on the dataset. It might be worth plotting learning curves to see whether the previous ensembles with fewer estimators were overfitting.