I am relatively new to machine learning in general, especially random forests.

Currently I am training a random regression forest with the scikit-learn library. My dataset has 500 features, the model produces a multi-output.

When training the model on 2 million and 3 million samples respectively, using n_estimators=3, max_depth=18, without touching the rest of the parameters, the saved model trained on 2 million samples is 50% larger than the one trained on 3 million samples. But not only the model size is larger, the accuracy is higher as well. This only happens, when I restrict the depth of the trees. When training the model without restricting the depth, then the accuracy increases with increasing number of samples.

I have read that deeper trees reduce the bias, while more trees reduce the variance. Does this consequently mean, that my dataset is somewhat biased and thus increasing the number of samples, increases the bias of the model, but this does not happen, when the depth is not restricted?

Is this behavior to be expected? Can anybody give me a hint to what is happening in the tree building process of RandomForestRegressor that results in this behavior?

If any additional information is needed to answer my question, please do not hesitate to ask in the comments.

EDIT1: I just fixed the seed as suggested by @RUser4512 and ran a test to reproduce the described behavior. All three test runs where trained with max_depth=15

n_samples=100,000: mean absolute error: 0.375; saved model size on disc: 4.6MB; Avg number of leaves per tree = 10,600

n_samples=200,000: mean absolute error: 0.312; saved model size on disc: 5.9MB; Avg number of leaves per tree = 12,610

n_samples=400,000: mean absolute error: 0.603; saved model size on disc: 1.3MB; Avg number of leaves per tree = 2,920

I thought about the nature of the prediction a little more. When doing inference, my model runs 2000 independent predictions which are later used to form a single final prediction. The 2000 single predictions have numerous outliers which are far from the ground truth. I am measuring the quality of a split with the criterion mean squared error, which penalizes outliers pretty heavily. Is there a chance, that this might be effected by increasing number of samples?

  • 1
    Setting n_estimators=3 means you're only training 3 trees. This is a very small number, and should be something more like 100. Given the randomness of each tree, the measured accuracy of the overall model could fluctuate wildly from run to run. So, it seems hard to read too much into the accuracy here. "The saved model trained on 2 million samples is 50% larger" What exactly are you measuring that's larger? – user20160 Jan 25 at 16:01
  • I trained the forests with the same set of parameters 3-5 times without any noticeable difference in accuracy. The accuracy definitely gets worse with increasing number of samples while the max_depth is fixed. "The saved model trained on 2 million samples is 50% larger", I dump my model to a .pkl file using joblib. So I just stated that the size of the saved model decreases in size with increasing number of samples. But it is to be noted, that this only happens after around 2million samples with max_depth=18. – ritocesura Jan 25 at 16:17

I have read that deeper trees reduce the bias, while more trees reduce the variance. Does this consequently mean, that my dataset is somewhat biased and thus increasing the number of samples, increases the bias of the model, but this does not happen, when the depth is not restricted?

No. "my dataset is somewhat biased" is an assertion that needs to be clarified. On the other hand, a biased estimator is a notion that is clearly defined (Wikipedia definition). Therefore, the bias of the random forest is not related to a kind of bias in the data.

But not only the model size is larger, the accuracy is higher as well

Regarding the issue about accuracy of your model decreasing with the number of sample, it is quite hard to know where it comes from.

Was the sampling performed randomly ? If so, did you fix the seed you use ? (both before sampling and before training the random forest) ? Though they are not parameters of the model, they can impact the accuracy you measure.

  • I sampled the n_samples used for training out of a pool with 16 million samples. No, I did not fix the seed for sampling the training data, nor did I fix the seed before training the random forest. I will train the model on 2 million and 3 million samples respectively, with a fixed seed and post the results here, as soon as I have them. – ritocesura Jan 26 at 13:04

I further inspected my dataset and found out that about 2% of the samples had very large outliers which were created in the process of generating the dataset. After removing those outliers not only did the MAE go way down, the above described behavior also did not reoccur.

If anybody is willing to explain why and how big outliers result in such behavior, I am happy to except their answer.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.