When to avoid Random Forest?

Question

Random forests are well known to perform fairly well on a variety of tasks and have been referred to as the leatherman of learning methods. Are there any types of problems or specific conditions in which one should avoid using a random forest?

Answer 1

Thinking about the specific language of the quotation, a leatherman is a multi-tool: a single piece of hardware with lots of little gizmos tucked into it. It's a pair of pliers, and a knife, and a screwdriver and more! Rather than having to carry each of these tools individually, the leatherman is a single item that you can clip to your trousers so it's always at hand. This is convenient, but the trade-off is that each of the component tools is not the best at its job. The can opener is hard to use, the screwdriver bits are usually the wrong size, and the knife can accomplish little more than whittling. If doing any of these tasks is critical, you'd be better served with a specialized tool: an actual knife, an actual screwdriver, or an actual pair of pliers.

A random forest can be thought of in the same terms. Random forest yields strong results on a variety of data sets, and is not incredibly sensitive to tuning parameters. But it's not perfect. The more you know about the problem, the easier it is to build specialized models to accommodate your particular problem.

There are a couple of obvious cases where random forests will struggle:

• Sparsity - When the data are very sparse, it's very plausible that for some node, the bootstrapped sample and the random subset of features will collaborate to produce an invariant feature space. There's no productive split to be had, so it's unlikely that the children of this node will be at all helpful. XGBoost can do better in this context.

• Data are not axis-aligned - Suppose that there is a diagonal decision boundary in the space of two features, $x_1$ and $x_2$. Even if this is the only relevant dimension to your data, it will take an ordinary random forest model many splits to describe that diagonal boundary. This is because each split is oriented perpendicular to the axis of either $x_1$ or $x_2$. (This should be intuitive because an ordinary random forest model is making splits of the form $x_1>4$.) Rotation forest, which performs a PCA projection on the subset of features selected for each split, can be used to overcome this: the projections into an orthogonal basis will, in principle, reduce the influence of the axis-aligned property because the splits will no longer be axis-aligned in the original basis.

  This image provides another example of how axis-aligned splits influence random forest decisions. The decision boundary is a circle at the origin, but note that this particular random forest model draws a box to approximate the circle. There are a number of things one could do to improve this boundary; the simplest include gathering more data and building more trees.

• Random forests basically only work on tabular data, i.e. there is not a strong, qualitatively important relationship among the features in the sense of the data being an image, or the observations being networked together on a graph. These structures are typically not well-approximated by many rectangular partitions. If your data live in a time series, or are a series of images, or live on a graph, or have some other obvious structure, the random forest will have a very hard time recognizing that. I have no doubt that researchers have developed variations on the method to attempt to accommodate these situations, but a vanilla random forest won't necessarily pick up on these structures in a helpful way. The good news is that you typically know when this is the case, i.e. you know you have images, a time-series or a graph to work with, so you can immediately apply a method more appropriate to that type of data.

Answer 2

Sharp corners. Exactness.

They use diffusion methods. They fit lumpy things well. They do not fit elaborate and highly detailed things well when the sample size is low. I would imagine that they do not do well on multivariate time-series data - when something over here depends on that one thing over there a distance.

Gradient boosted forests might fit or over-fit, but can get substantially lower error for the same data.

"Leathermen" do not exist. There are no "silver bullets". There are toolboxes. Know your tools, and take good care of them so they can take care of you. Be wary of "when you are a hammer, then every problem looks like a nail" especially when you do not have a dense library in your toolbox.

Until you know the problem well, it is easy to imagine anything might solve it, or your favorite tool might solve it. Wisdom suggests getting deep in understanding the problem, and being very familiar with your tools.

Added: If you have enough compute resources or time margin to use something else. The RF is not only fast to train, but fast to execute. A very deep boosted structure is less of that. You have to have the overhead to support that.

Answer 3

This is the first time I actually answer a question, so do not pin me down on it .. but I do think I can answer your question:

If you are indeed only interested in model performance and not in thing like interpretability, random forest is indeed often a very good learning algorithm, but does perform slightly worse in the following cases:

1.) When the dimensionality (number of features) is very high with respect to the number of training samples, in those cases a regularized linear regression or SVM would be better.

2.) In the case there are higher order representations/convolutional structures in the data, like e.g. in computer vision problems. In those computer vision cases a convolutional neural network will outperform a random forest (In general if there is knowledge one can incorporate into the learning that is a better thing).

That being said random forest are a very good starting point. One of the person I admire for his Machine Learning skills always starts with learning a random forest and a regularized linear regressor.

However, if you want the best possible performance I believe nowadays neural networks aka. Deep Learning is looking like a very attractive approach. More and more winners on data-challenge websites like Kaggle use Deep Learning models for the competition. Another pro of neural networks is that they can handle very large numbers of samples (>10^6 one can train them using stochastic gradient descend, feeding bits of data at a time). Personally I find this a very attractive pro for Deep Learning.

Answer 4

Random forest fits multi-dimensional staircase to your data. It produces sharp edges in predictions. If your data are of continuous nature, then probably there are better methods to fit them. That doesn't mean however that "you should avoid random forest to fit them" :-) I don't think there is any kind of data where you "should be worried" about using RF.

And, I wouldn't say RF is "leatherman", it's definitely not, in the sense that "it will do everything for you". It's just a basic universal method you can use on wide variety of data. I would say it's more like "first simple thing to try", or a "baseline" for benchmarking better methods. I also use it when I want to see which predictor variables are important; there is an excellent R package Boruta for this purpose.

Answer 5

There's already many good points by others e.g. about sparse feature spaces, when we know a step-function will not work well/when you don't have a good representation of the data (some of these may of course be a matter of creating better features first), as well as non-tabular data types where other approaches are known to work better especially for how to represent the input data (as mentioned by others: images, text, audio).

However, here are some additional situations that are non-ideal for RF (in approximately descending order of my strength of concern about using RF). Several of these are not specifically an issue with RF, but rather apply more broadly to many prediction modelling approaches.

• When we know mechanistically what underlies the system we model. E.g. you can often define a set of biologically sensible differential equations that govern the blood levels of a drug in the human body. You are much better off to set-up a non-linear model based on these, while any modelling approach that ignores this understanding of the biology is going to be a lot less efficient - especially for small datasets -, and likely terrible at extrapolation (see below). I would not ever seriously consider basic RF (or other "default" prediction models like gradient boosted decision trees) here.
• When you need to extrapolate beyond your training data. The fitted step-functions will remain constant outside the training feature value range, which will often be an implausible thing to assume. Extrapolation is generally a big challenge, but e.g. mechanistic models from the previous bullet point would be a much, much better bet for extrapolation than RF.
• RF is also not so great for multivariate outputs. Even something as simple as many correlated multi-label binary classification tasks can be more efficient with neural networks. It becomes even clearer, when the desired outputs is e.g. an image, a series of values such as a sentence of words, or a variable number of bounding boxes with class assignments.
• When you want to causally interpret the effect of a feature. This may be a really silly and obvious thing to point out, but I've seen too many people saying things like:"My RF predicts that being in a hospital is a predictor of a higher risk of death! Don't go to the hospital, if you get sick!" or similar other misinterpretations. Obviously, you need a different approach than the standard prediction set-up in order to try to approach a causal question (for which there are approaches that try to use RF).
• When there are high cardinality categorical features. E.g. user ID when there's hundreds or thousands of users (in a sense see sparse inputs), products that are being sold etc. and so on. This is not necessarily a case where RF should not be used, but rather where other approaches (such as neural networks with embeddings) might outperform them (of course, you could use neural network embeddings as inputs).
• When you need to win a data science competition on tabular data, RF is also not necessarily the first thing you'd try. Gradient boosting (xgboost, LightGBM, catboost etc.) tends to outperform RF once properly tuned. It's not necessarily by much and may not be of relevance for many applications unless every little improvement in prediction performance matters.

Answer 6

RF is a static model and when it comes to dealing with dynamical systems and online learning, the cost of adapting to the data drift is the same as reconstructing the entire structure again.

Answer 7

First of all, the Random Forest cannot be applied to the following data types:

• images
• audio
• text (after preprocessing data will be sparse and RF doesn't work well with sparse data)

For tabular data type, it is always good to check Random Forest because:

• it requires less data preparation and preprocessing than Neural Networks or SVMs. For example, you don't need to do feature scaling.
• For Random Forest training you can just use default parameters and set the number of trees (the more trees in RF the better). When you compare Random Forest to Neural Networks, the training is very easy (don't need to define architecture, or tune training algorithm). Random Forest is easier to train than Neural Networks.

Answer 8

Random Forests are prone to exhausing memory and causing out-of-memory errors (as compared to an incremental / batch learning method that fixes memory usage.