Random forests are well known to perform fairly well on a variety of tasks and have been referred to as the leatherman of learning methods. Are there any types of problems or specific conditions in which one should avoid using a random forest?

  • 8
    $\begingroup$ Hi. "PLS is the leatherman of ...", "Bootstrap is the leatherman of ...", "Random forest is the leatherman of ..." <- I advise you to be suspicious about such claims. It was just a comment :) $\endgroup$ – Stéphane Laurent Aug 16 '14 at 21:00

Thinking about the specific language of the quotation, a leatherman is a multi-tool: a single piece of hardware with lots of little gizmos tucked into it. It's a pair of pliers, and a knife, and a screwdriver and more! Rather than having to carry each of these tools individually, the leatherman is a single item that you can clip to your trousers so it's always at hand. This is convenient, but the trade-off is that each of the component tools is not the best at its job. The can opener is hard to use, the screwdriver bits are usually the wrong size, and the knife can accomplish little more than whittling. If doing any of these tasks is critical, you'd be better served with a specialized tool: an actual knife, an actual screwdriver, or an actual pair of pliers.

A random forest can be thought of in the same terms. Random forest yields strong results on a variety of data sets, and is not incredibly sensitive to tuning parameters. But it's not perfect. The more you know about the problem, the easier it is to build specialized models to accommodate your particular problem.

There are a couple of obvious cases where random forests will struggle:

  • Sparsity - When the data are very sparse, it's very plausible that for some node, the bootstrapped sample and the random subset of features will collaborate to produce an invariant feature space. There's no productive split to be had, so it's unlikely that the children of this node will be at all helpful. XGBoost can do better in this context.

  • Data are not axis-aligned - Suppose that there is a diagonal decision boundary in the space of two features, $x_1$ and $x_2$. Even if this is the only relevant dimension to your data, it will take an ordinary random forest model many splits to describe that diagonal boundary. This is because each split is oriented perpendicular to the axis of either $x_1$ or $x_2$. (This should be intuitive because an ordinary random forest model is making splits of the form $x_1>4$.) Rotation forest, which performs a PCA projection on the subset of features selected for each split, can be used to overcome this: the projections into an orthogonal basis will, in principle, reduce the influence of the axis-aligned property because the splits will no longer be axis-aligned in the original basis.

    This image provides another example of how axis-aligned splits influence random forest decisions. The decision boundary is a circle at the origin, but note that this particular random forest model draws a box to approximate the circle. There are a number of things one could do to improve this boundary; the simplest include gathering more data and building more trees. enter image description here

  • Random forests basically only work on tabular data, i.e. there is not a strong, qualitatively important relationship among the features in the sense of the data being an image, or the observations being networked together on a graph. These structures are typically not well-approximated by many rectangular partitions. If your data live in a time series, or are a series of images, or live on a graph, or have some other obvious structure, the random forest will have a very hard time recognizing that. I have no doubt that researchers have developed variations on the method to attempt to accommodate these situations, but a vanilla random forest won't necessarily pick up on these structures in a helpful way. The good news is that you typically know when this is the case, i.e. you know you have images, a time-series or a graph to work with, so you can immediately apply a method more appropriate to that type of data.
| cite | improve this answer | |
  • $\begingroup$ I think vanilla random forests have the potential to recognise non rectangular partitions. We can have quadratic features for example and recognise boundaries like x^2 < c. $\endgroup$ – Aniruddha Acharya Sep 10 '16 at 18:18
  • 2
    $\begingroup$ @AniruddhaAcharya I said "struggle," not fail. My answer here makes it clear that quadratic features are not an insurmountable problem. But diagonals or quadratics or other non-aligned types of splits will require the trees to split on those features again and again in a rectangular fashion to approximate a diagonal/curve. You can see that effect in the plot to this post: look at the sharp corners. RF is using a square to approximate a circle. $\endgroup$ – Sycorax Sep 10 '16 at 18:33
  • $\begingroup$ I meant feeding the quadratic (and other higher order) features to RF would reduce the struggle in modelling non rectangular partitions. For example RFs can use 2 quadratics to approximate the circle instead of using a square. Although I agree that its not as easy as models that incorporate interaction between variables, I feel its not that hard because RFs are not restricted to using thresholds on the raw features. $\endgroup$ – Aniruddha Acharya Sep 11 '16 at 2:09
  • 1
    $\begingroup$ @AniruddhaAcharya If you know that square terms are the right model, you don't need random forest: just add square terms to a linear model. RF is a good way to approximate arbitrary functions, especially when they have axis-aligned decision functions. $\endgroup$ – Sycorax Sep 11 '16 at 4:55
  • $\begingroup$ Why is the (quadratic terms) feature engineering suggested by @AniruddhaAcharya not a useful step to use in conjunction with RF's? Yes those features could be added to a linear model: but the latter does not provide the behavior of combining the contributions of many results that are individually optimized for differing subsets of the features. $\endgroup$ – javadba Jan 1 '18 at 18:16

Sharp corners. Exactness.

They use diffusion methods. They fit lumpy things well. They do not fit elaborate and highly detailed things well when the sample size is low. I would imagine that they do not do well on multivariate time-series data - when something over here depends on that one thing over there a distance.

Gradient boosted forests might fit or over-fit, but can get substantially lower error for the same data.

"Leathermen" do not exist. There are no "silver bullets". There are toolboxes. Know your tools, and take good care of them so they can take care of you. Be wary of "when you are a hammer, then every problem looks like a nail" especially when you do not have a dense library in your toolbox.

Until you know the problem well, it is easy to imagine anything might solve it, or your favorite tool might solve it. Wisdom suggests getting deep in understanding the problem, and being very familiar with your tools.

Added: If you have enough compute resources or time margin to use something else. The RF is not only fast to train, but fast to execute. A very deep boosted structure is less of that. You have to have the overhead to support that.

| cite | improve this answer | |
  • 2
    $\begingroup$ To be the devil's advocate here: Nothing will "fit elaborate and highly detailed things well when the sample size is low." $\endgroup$ – usεr11852 Jan 25 '19 at 21:38

This is the first time I actually answer a question, so do not pin me down on it .. but I do think I can answer your question:

If you are indeed only interested in model performance and not in thing like interpretability random forest are indeed often a very good learning algorithm, but do perform slightly worse in the following cases:

1.) When the dimensionality (number of features) is very high with respect to the number of training samples, in those cases a regularized linear regression or SVM would be better.

2.) In the case there are higher order representations/convolutional structures in the data, like e.g. in computer vision problems. In those computer vision cases a convolutional neural network will outperform a random forest (In general if there is knowledge one can incorporate into the learning that is a better thing).

That being said random forest are a very good starting point. One of the person I admire for his Machine Learning skills always starts with learning a random forest and a regularized linear regressor.

However, if you want the best possible performance I believe nowadays neural networks aka. Deep Learning is looking like a very attractive approach. More and more winners on data-challenge websites like Kaggle use Deep Learning models for the competition. Another pro of neural networks is that they can handle very large numbers of samples (>10^6 one can train them using stochastic gradient descend, feeding bits of data at a time). Personally I find this a very attractive pro for Deep Learning.

| cite | improve this answer | |
  • 2
    $\begingroup$ Nice answer, but your last point is not correct: mini-batch training can be implemented for all common machine learning algorithms, see for example h2o. $\endgroup$ – jubo Aug 16 '14 at 21:48
  • $\begingroup$ Oke, interesting, I did not know that. Are you saying that using these methods one can train a (decision) tree using mini-batch/SGD learning and thus build a single tree based on a majority split of the training total data (say 65%)? $\endgroup$ – MJW Aug 17 '14 at 11:59
  • 1
    $\begingroup$ I think random forest still should be good when the number of features is high - just don't use a lot of features at once when building a single tree, and at the end you'll have a forest of independent classifiers that collectively should (hopefully) do well. $\endgroup$ – Alexey Grigorev Aug 17 '14 at 16:06
  • $\begingroup$ As far as I understand, h2o uses the map-reduce paradigm for minibatch-training. Single (CART) trees are not implemented as such in h2o (but I suppose an unpruned tree is a special case of random forest with just one tree and maximum choice of predictors?). $\endgroup$ – jubo Aug 17 '14 at 18:07
  • 2
    $\begingroup$ What is the justification for (1)? You make a claim, but don't provide any supporting argument. $\endgroup$ – Sycorax Jan 1 '18 at 19:32

First of all, the Random Forest cannot be applied to the following data types:

  • images
  • audio
  • text (after preprocessing data will be sparse and RF doesn't work well with sparse data)

For tabular data type, it is always good to check Random Forest because:

  • it requires less data preparation and preprocessing than Neural Networks or SVMs. For example, you don't need to do feature scaling.
  • For Random Forest training you can just use default parameters and set the number of trees (the more trees in RF the better). When you compare Random Forest to Neural Networks, the training is very easy (don't need to define architecture, or tune training algorithm). Random Forest is easier to train than Neural Networks.
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.