Okay, I have created a Random Forest, I get a very good prediction of response. What's next? How can I improve it further?

I ask this because I often see competitions in Kaggle, where RF is used as a benchmark, and many people manage to obtain scores much higher than that of the RF based solution. In some cases, the data is also unlabelled, preventing you to apply manual judgment.

How do they beat RF by such decent margins?


2 Answers 2


The key is to build more models. If read solutions papers for competitions like the Netflix prize, the winning team had something like 100+ models doing the prediction. They then combined the models linearly (i.e. regress the actual values against the 100 or so predicted values) to get weights for individual models. There were maybe four or five different general types of models, and then the rest were variations on those models with different features or with different tuning parameters.

Though it's important to note that each additional model gives you significantly decreasing returns. I think the winners of the Netflix prize found that 16 models got them to within a few percent of their final RMSE, meaning that the other ~80 some models contributed tinier and tinier amounts of improvement to the RMSE.

  • $\begingroup$ Nice response, I have been looking for a while for tutorials on how to calibrate Random Forests. I have seen many discussions on this topic on Kaggle but never a concrete example/code. You could use logistic regression I think, lasso? even elastic net as is mentioned in the elements of statistical learning. But I am not quite sure on how to actually do it. Do you have an actual example of this? It would be deeply appreciated! $\endgroup$
    – JEquihua
    Sep 19, 2013 at 4:30
  • $\begingroup$ If you are using R, you should check out the caret package: caret.r-forge.r-project.org. It basically automates the tuning process for a lot of common models in R. The package's webpage and its documentation have walkthroughs. But it is still more of the "how" rather than the intuition and art of model tuning. Unfortunately I too am still trying to learn the latter, myself. Could just be a lack of experience thing, in which case, fake it till you make it :-) $\endgroup$
    – hgcrpd
    Sep 20, 2013 at 9:51

RF can be tune by different parameters. The most important parameter is the number of variables randomly sampled. The author recommend different values for regression and classification. But it is worth to tune by your own problem and data set.

The size of terminal nodes of trees may also affect the performance. Similar to above, there are default values recommended by the author, but we can tune it.

The number of trees may be tuned if needed, but RF is seldom overfitted. So this may not affect too much if the number of trees is large enough.

Well, different model has different advantages and different strong application which they are suitable. This depends on the characteristics of your data set, some other special models are likely to better than RF in some specific topics. But RF is easy to tune, and it is usually reasonably good model, generally speaking, even by default setting.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.