1
$\begingroup$

The first time I used Random Forest, it gave me an accuracy of 81% with these parameters: n_estimators:10, max_features: sqrt(20), min_samples_split: 2 and min_samples_leaf: 1

Then I decided to optimize.

1) I did a random search. Accuracy: 89.76%, n_estimators: 10, max_features:8, min_samples_split:6, min_samples_leaf:1. This took: 19 minutes.

2) Then I read that it is a good idea to uses OOB error to find optimal values for n_estimators and max_features. So I did it. The plot showed that with less than 100 trees the OOB error is high and inestable. And that max_features: 4 to 10 is good.

3) So I ran once more the random search. But I specified to look for trees over 100. The best parameters are: n_ estimators: 127, max_features: 7, min_samples_split: 5 and min_samples_leaf: 1. Accuracy: 91,56%. Time: 6 hours!!!!!

As you can see the increase in accuracy is 3% more. But it took 6 hours.

So is it worth increasing the number of trees(n_estimators) from 10 to 127?

$\endgroup$
1
  • $\begingroup$ You probably should always use the maximum number of trees you can afford, there's no downside outside training time. $\endgroup$
    – Firebug
    Dec 24, 2016 at 15:12

1 Answer 1

1
$\begingroup$

Breiman said in his original papers in Machine Learning journal, to not be "stingy" and use 5,000 to 50,000 trees -- but he had a lot of computational power. Recall, that for RF, the in-bag bootstrapped objects are only used for training the trees, not testing. Testing (i.e. error) is determined when the out-of-bag (OOB) objects are dropped down the trained trees, after which error is based on the percentage of class purity in the final nodes when node-splitting can no longer be performed. Importance scores are also based on permuting feature-based columns in the OOB datasets, which gives loss of predictive accuracy as a function of each feature.

I never use less than 1,000 trees for any RF runs. Also, the default number of features randomly selected for node-splitting in a tree is typically srqt(total # of features). You probably don't need to use the 6-hour run approach since you are hunting down better accuracy. RF is one of the better classifiers, whose default parameter usage should do fine in terms of bias/variance and robustness across different data sets. Recall, RFs generalize better than other classifiers, so the increased generalization comes at a cost of lower accuracy -- I am always comfortable with 75%-80% accuracy from RF, while other classifiers can be 90-95% accurate. When you launch an ensemble of classifiers into the real world, you can always rest assured that the RF will probably do better than any of the other classifiers. It won't get faked out, and probably won't break down, etc.

$\endgroup$
2
  • $\begingroup$ When you say "I am fine with 75%-80% accuracy" do you mean on the training data? Does it really even matter what the accuracy on the training data is, I'd argue it is best to ignore it completely. $\endgroup$ Dec 24, 2016 at 17:29
  • $\begingroup$ You don't know what the class predictive accuracy is for the training objects that are in the in-bag samples, since it is never determined. Accuracy only comes from the OOB objects that are dropped down trained trees. It's almost the same for CV, where you only predict class of test objects in fold(s) whose objects are not used for training. $\endgroup$
    – user32398
    Dec 24, 2016 at 17:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.