29
$\begingroup$

Random forests are well known to perform fairly well on a variety of tasks and have been referred to as the leatherman of learning methods. Are there any types of problems or specific conditions in which one should avoid using a random forest?

$\endgroup$
2
  • 9
    $\begingroup$ Hi. "PLS is the leatherman of ...", "Bootstrap is the leatherman of ...", "Random forest is the leatherman of ..." <- I advise you to be suspicious about such claims. It was just a comment :) $\endgroup$ – Stéphane Laurent Aug 16 '14 at 21:00
  • $\begingroup$ Random forest fits multi-dimensional staircase to your data. It produces sharp edges in predictions. If your data are of continuous nature, then probably there are better methods to fit them. That doesn't mean however that "you should avoid random forest to fit them" :-) I don't think there is any kind of data where you "should be worried" about using RF. $\endgroup$ – Innate Imunity is The Way Feb 3 at 14:42
30
$\begingroup$

Thinking about the specific language of the quotation, a leatherman is a multi-tool: a single piece of hardware with lots of little gizmos tucked into it. It's a pair of pliers, and a knife, and a screwdriver and more! Rather than having to carry each of these tools individually, the leatherman is a single item that you can clip to your trousers so it's always at hand. This is convenient, but the trade-off is that each of the component tools is not the best at its job. The can opener is hard to use, the screwdriver bits are usually the wrong size, and the knife can accomplish little more than whittling. If doing any of these tasks is critical, you'd be better served with a specialized tool: an actual knife, an actual screwdriver, or an actual pair of pliers.

A random forest can be thought of in the same terms. Random forest yields strong results on a variety of data sets, and is not incredibly sensitive to tuning parameters. But it's not perfect. The more you know about the problem, the easier it is to build specialized models to accommodate your particular problem.

There are a couple of obvious cases where random forests will struggle:

  • Sparsity - When the data are very sparse, it's very plausible that for some node, the bootstrapped sample and the random subset of features will collaborate to produce an invariant feature space. There's no productive split to be had, so it's unlikely that the children of this node will be at all helpful. XGBoost can do better in this context.

  • Data are not axis-aligned - Suppose that there is a diagonal decision boundary in the space of two features, $x_1$ and $x_2$. Even if this is the only relevant dimension to your data, it will take an ordinary random forest model many splits to describe that diagonal boundary. This is because each split is oriented perpendicular to the axis of either $x_1$ or $x_2$. (This should be intuitive because an ordinary random forest model is making splits of the form $x_1>4$.) Rotation forest, which performs a PCA projection on the subset of features selected for each split, can be used to overcome this: the projections into an orthogonal basis will, in principle, reduce the influence of the axis-aligned property because the splits will no longer be axis-aligned in the original basis.

    This image provides another example of how axis-aligned splits influence random forest decisions. The decision boundary is a circle at the origin, but note that this particular random forest model draws a box to approximate the circle. There are a number of things one could do to improve this boundary; the simplest include gathering more data and building more trees. enter image description here

  • Random forests basically only work on tabular data, i.e. there is not a strong, qualitatively important relationship among the features in the sense of the data being an image, or the observations being networked together on a graph. These structures are typically not well-approximated by many rectangular partitions. If your data live in a time series, or are a series of images, or live on a graph, or have some other obvious structure, the random forest will have a very hard time recognizing that. I have no doubt that researchers have developed variations on the method to attempt to accommodate these situations, but a vanilla random forest won't necessarily pick up on these structures in a helpful way. The good news is that you typically know when this is the case, i.e. you know you have images, a time-series or a graph to work with, so you can immediately apply a method more appropriate to that type of data.
$\endgroup$
10
  • $\begingroup$ I think vanilla random forests have the potential to recognise non rectangular partitions. We can have quadratic features for example and recognise boundaries like x^2 < c. $\endgroup$ – Aniruddha Acharya Sep 10 '16 at 18:18
  • 2
    $\begingroup$ @AniruddhaAcharya I said "struggle," not fail. My answer here makes it clear that quadratic features are not an insurmountable problem. But diagonals or quadratics or other non-aligned types of splits will require the trees to split on those features again and again in a rectangular fashion to approximate a diagonal/curve. You can see that effect in the plot to this post: look at the sharp corners. RF is using a square to approximate a circle. $\endgroup$ – Sycorax Sep 10 '16 at 18:33
  • $\begingroup$ I meant feeding the quadratic (and other higher order) features to RF would reduce the struggle in modelling non rectangular partitions. For example RFs can use 2 quadratics to approximate the circle instead of using a square. Although I agree that its not as easy as models that incorporate interaction between variables, I feel its not that hard because RFs are not restricted to using thresholds on the raw features. $\endgroup$ – Aniruddha Acharya Sep 11 '16 at 2:09
  • 1
    $\begingroup$ @AniruddhaAcharya If you know that square terms are the right model, you don't need random forest: just add square terms to a linear model. RF is a good way to approximate arbitrary functions, especially when they have axis-aligned decision functions. $\endgroup$ – Sycorax Sep 11 '16 at 4:55
  • $\begingroup$ Why is the (quadratic terms) feature engineering suggested by @AniruddhaAcharya not a useful step to use in conjunction with RF's? Yes those features could be added to a linear model: but the latter does not provide the behavior of combining the contributions of many results that are individually optimized for differing subsets of the features. $\endgroup$ – StephenBoesch Jan 1 '18 at 18:16
23
$\begingroup$

Sharp corners. Exactness.

They use diffusion methods. They fit lumpy things well. They do not fit elaborate and highly detailed things well when the sample size is low. I would imagine that they do not do well on multivariate time-series data - when something over here depends on that one thing over there a distance.

Gradient boosted forests might fit or over-fit, but can get substantially lower error for the same data.

"Leathermen" do not exist. There are no "silver bullets". There are toolboxes. Know your tools, and take good care of them so they can take care of you. Be wary of "when you are a hammer, then every problem looks like a nail" especially when you do not have a dense library in your toolbox.

Until you know the problem well, it is easy to imagine anything might solve it, or your favorite tool might solve it. Wisdom suggests getting deep in understanding the problem, and being very familiar with your tools.

Added: If you have enough compute resources or time margin to use something else. The RF is not only fast to train, but fast to execute. A very deep boosted structure is less of that. You have to have the overhead to support that.

$\endgroup$
2
  • 4
    $\begingroup$ To be the devil's advocate here: Nothing will "fit elaborate and highly detailed things well when the sample size is low." $\endgroup$ – usεr11852 Jan 25 '19 at 21:38
  • $\begingroup$ @usεr11852 - I strongly agree. Nothing is good at it. Some things are less bad than others. I like where zero-sample learning is going there. $\endgroup$ – EngrStudent Feb 3 at 15:20
5
$\begingroup$

This is the first time I actually answer a question, so do not pin me down on it .. but I do think I can answer your question:

If you are indeed only interested in model performance and not in thing like interpretability, random forest is indeed often a very good learning algorithm, but does perform slightly worse in the following cases:

1.) When the dimensionality (number of features) is very high with respect to the number of training samples, in those cases a regularized linear regression or SVM would be better.

2.) In the case there are higher order representations/convolutional structures in the data, like e.g. in computer vision problems. In those computer vision cases a convolutional neural network will outperform a random forest (In general if there is knowledge one can incorporate into the learning that is a better thing).

That being said random forest are a very good starting point. One of the person I admire for his Machine Learning skills always starts with learning a random forest and a regularized linear regressor.

However, if you want the best possible performance I believe nowadays neural networks aka. Deep Learning is looking like a very attractive approach. More and more winners on data-challenge websites like Kaggle use Deep Learning models for the competition. Another pro of neural networks is that they can handle very large numbers of samples (>10^6 one can train them using stochastic gradient descend, feeding bits of data at a time). Personally I find this a very attractive pro for Deep Learning.

$\endgroup$
5
  • 2
    $\begingroup$ Nice answer, but your last point is not correct: mini-batch training can be implemented for all common machine learning algorithms, see for example h2o. $\endgroup$ – jubo Aug 16 '14 at 21:48
  • $\begingroup$ Oke, interesting, I did not know that. Are you saying that using these methods one can train a (decision) tree using mini-batch/SGD learning and thus build a single tree based on a majority split of the training total data (say 65%)? $\endgroup$ – MJW Aug 17 '14 at 11:59
  • 1
    $\begingroup$ I think random forest still should be good when the number of features is high - just don't use a lot of features at once when building a single tree, and at the end you'll have a forest of independent classifiers that collectively should (hopefully) do well. $\endgroup$ – Alexey Grigorev Aug 17 '14 at 16:06
  • $\begingroup$ As far as I understand, h2o uses the map-reduce paradigm for minibatch-training. Single (CART) trees are not implemented as such in h2o (but I suppose an unpruned tree is a special case of random forest with just one tree and maximum choice of predictors?). $\endgroup$ – jubo Aug 17 '14 at 18:07
  • 2
    $\begingroup$ What is the justification for (1)? You make a claim, but don't provide any supporting argument. $\endgroup$ – Sycorax Jan 1 '18 at 19:32
2
$\begingroup$

Random forest fits multi-dimensional staircase to your data. It produces sharp edges in predictions. If your data are of continuous nature, then probably there are better methods to fit them. That doesn't mean however that "you should avoid random forest to fit them" :-) I don't think there is any kind of data where you "should be worried" about using RF.

And, I wouldn't say RF is "leatherman", it's definitely not, in the sense that "it will do everything for you". It's just a basic universal method you can use on wide variety of data. I would say it's more like "first simple thing to try", or a "baseline" for benchmarking better methods. I also use it when I want to see which predictor variables are important; there is an excellent R package Boruta for this purpose.

$\endgroup$
1
$\begingroup$

There's already many good points by others e.g. about sparse feature spaces, when we know a step-function will not work well/when you don't have a good representation of the data (some of these may of course be a matter of creating better features first), as well as non-tabular data types where other approaches are known to work better especially for how to represent the input data (as mentioned by others: images, text, audio).

However, here are some additional situations that are non-ideal for RF (in approximately descending order of my strength of concern about using RF). Several of these are not specifically an issue with RF, but rather apply more broadly to many prediction modelling approaches.

  • When we know mechanistically what underlies the system we model. E.g. you can often define a set of biologically sensible differential equations that govern the blood levels of a drug in the human body. You are much better off to set-up a non-linear model based on these, while any modelling approach that ignores this understanding of the biology is going to be a lot less efficient - especially for small datasets -, and likely terrible at extrapolation (see below). I would not ever seriously consider basic RF (or other "default" prediction models like gradient boosted decision trees) here.
  • When you need to extrapolate beyond your training data. The fitted step-functions will remain constant outside the training feature value range, which will often be an implausible thing to assume. Extrapolation is generally a big challenge, but e.g. mechanistic models from the previous bullet point would be a much, much better bet for extrapolation than RF.
  • RF is also not so great for multivariate outputs. Even something as simple as many correlated multi-label binary classification tasks can be more efficient with neural networks. It becomes even clearer, when the desired outputs is e.g. an image, a series of values such as a sentence of words, or a variable number of bounding boxes with class assignments.
  • When you want to causally interpret the effect of a feature. This may be a really silly and obvious thing to point out, but I've seen too many people saying things like:"My RF predicts that being in a hospital is a predictor of a higher risk of death! Don't go to the hospital, if you get sick!" or similar other misinterpretations. Obviously, you need a different approach than the standard prediction set-up in order to try to approach a causal question (for which there are approaches that try to use RF).
  • When there are high cardinality categorical features. E.g. user ID when there's hundreds or thousands of users (in a sense see sparse inputs), products that are being sold etc. and so on. This is not necessarily a case where RF should not be used, but rather where other approaches (such as neural networks with embeddings) might outperform them (of course, you could use neural network embeddings as inputs).
  • When you need to win a data science competition on tabular data, RF is also not necessarily the first thing you'd try. Gradient boosting (xgboost, LightGBM, catboost etc.) tends to outperform RF once properly tuned. It's not necessarily by much and may not be of relevance for many applications unless every little improvement in prediction performance matters.
$\endgroup$
0
$\begingroup$

First of all, the Random Forest cannot be applied to the following data types:

  • images
  • audio
  • text (after preprocessing data will be sparse and RF doesn't work well with sparse data)

For tabular data type, it is always good to check Random Forest because:

  • it requires less data preparation and preprocessing than Neural Networks or SVMs. For example, you don't need to do feature scaling.
  • For Random Forest training you can just use default parameters and set the number of trees (the more trees in RF the better). When you compare Random Forest to Neural Networks, the training is very easy (don't need to define architecture, or tune training algorithm). Random Forest is easier to train than Neural Networks.
$\endgroup$
0
$\begingroup$

Random Forests are prone to exhausing memory and causing out-of-memory errors (as compared to an incremental / batch learning method that fixes memory usage.

$\endgroup$
2
  • $\begingroup$ I don't agree, there are methods which consume much more memory. Never had memory problems with RF. $\endgroup$ – Innate Imunity is The Way Feb 3 at 14:39
  • 1
    $\begingroup$ Is there a sense in which exhausting memory is unique or particularly acute in random forests, but not a problem for other model types? Incremental/batch learning is only more memory efficient because it reduces the amount of the training data stored in main memory. In other words, the memory problem that batch learning solves is generic to any dataset that is too large, not a problem with random forest itself. You can also train random forests by subsetting the data and training 1 tree on a batch, only storing the batch in memory, so this claim isn't even true for random forests. $\endgroup$ – Sycorax Feb 3 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.