What is the intuition behind why randomForest works well with prediction on lots of applied datasets? No fancy math required for an answer-but if you have some that helps intuition great.

  • 4
    $\begingroup$ The mean. And bootstrapping makes it a mean of means. And the ensemble turns a weak learner into a robust one. $\endgroup$ – EngrStudent Dec 29 '17 at 0:07
  • $\begingroup$ @EngrStudent how is the prediction from a single decision tree "a mean"? $\endgroup$ – Lucas Roberts Dec 30 '17 at 15:56
  • 1
    $\begingroup$ It is an analog to the mean, a "high-tech big brother", or "mutant offspring". You could fit a "stump" to the data and each of the two branches of the stump would be a single mean. The bagging makes the dimension and location of the "split", and the value of the mean fuzzier, so they can be inexact, more a representation of a central behavior, and more robust. $\endgroup$ – EngrStudent Dec 30 '17 at 22:32
  • $\begingroup$ @EngrStudent, I'm still not sure I understand. Also I'm still not sure how this provides any intuition. Could you elaborate? $\endgroup$ – Lucas Roberts Dec 31 '17 at 21:12

The kind of intuition I like to have is driven by "physics" and connects to "first principles".

You can think of the random forest is a machine, like a flour mill. You put something in, and you get something else out. The two ways to gain intuition are to "look at the output grain" and look at how the machine is put together (assembled).

Going into full depth would take a book. This is a few paragraphs, so it is going to be a relatively shallow exploration.

This is about the a single dust of flour:
If you must bet on a horse race, and you know as little about formal/historic betting as I do, then you might look at the mean times for each of the last several races of all the horses and bet that one that has higher consistent performance than some others is likely to outperform those others. This is a very simple mean. It doesn't account for other factors, but in the long run, it is likely to be a decent predictor of typical performance. You would lose less money than if you uniformly randomly picked a horse and bet on it to win. In this sense, in the long run, the mean can be thought of as the "best" one-parameter estimator of "typical" performance. The "learner" is a single number that doesn't change, but it has some power.

If we instead trained a 2-parameter estimator on each horse, you might have one "constant" parameter and one "time-varying" parameter. This would be a simple linear model.

Now the constant-time estimation of horse race has a number of flaws. Performance changes. Horses get older. Some get sick. There is doping, which is illegal. I would guess that a familiar rider might get more performance than an unfamiliar one.

In the grand scheme of things, accounting for those sources of variation is going to give much better predictive accuracy. The basic mean is going to do that relatively poorly in the grand scheme, but in very small parcels (boxy regions of the input space with volume and coordinates, sometimes also called voxels) it can do fairly well.

And that is a "grain of dust" of a classification and regression tree (CART). Within a box, the model is typically either a mean, or a linear function.

This is about how the model is put together:
Now for the second problem. The right small parcel isn't easy to find: it is easy to get wrong. Typical analytic methods aren't very robust and finding the "edges" where the basic model breaks down. The "entropy" or "variance" methods are related to the square of the derivative, and given noise and sparse sampling, these tend to be easier to estimate wrongly. The parallel "ensemble", using many CART trees with uniformly random different training inputs, then averaging the array of outputs to find the "mean" edge, gives better results. It turns out that in the process of finding the "edge" we also get multiple, slightly different, overlapping models for the "interior function" so we get a better central estimate for a better box.

Bottom lines:
So in its atomic form, in its soul it is a central tendency or a line. The way the bounds for this "atom" are found, using the parallel ensemble, overcomes the weakness of the second derivative of noise in data, and is another way of thinking of a mean: the mean of bootstrapped models.

(This makes me think a multivariate, nonuniform, Savitzky-Golay estimator might also be useful for finding "splits" for branches in CART models. update: we don't need multivariate, or non-uniform. A fun consequence is that we can split at a partial value that is data-driven using an approach analogous to sub-pixel image registration. Because it is frequency-domain, it should be relatively fast as well. update2: I like how a 5 or 7 element filter can strongly indicate variance. New SO Question.)

There are some places where a random forest is not ideal. link. Personally, I think that when you apply it where it was meant to go, it cannot be beat. I think that it is about central tendency, not high accuracy. I think that it is very fast computationally, and handles sometimes very high dimensional data relatively well. There are no "silver bullets", no "leathermen".

One of the ways I like to treat an RF, is as an extended version of the mean as follows. In many machine learning applications you center and scale your data before you put it into the learner. The idea is for the high-accuracy tool to handle the hard parts, and leave the easy parts to easy tools.

  • $\begingroup$ -> if you are using CART trees they are invariant to monotone transforms of the covariate space so I'm not sure I understand your last comment. The CART regression tree variant is a mean, but the classification tree variety is a estimate of the most frequent class in the terminal node. Your explanation might hold for certain specific implementations but not more generally-I'm still confused by your explanation. $\endgroup$ – Lucas Roberts Jan 3 '18 at 14:31
  • $\begingroup$ I mean it in the opposite way to what you seem to be thinking. I make y_hat_1 equal to y - lm(x). I then fit a random forest to y_hat_1 and make y_hat2 equal y_hat_1 - rf(x). I then fit y_hat_2 with a gbm, make an estimate y_hat_2_hat = gbm(x) and then I combine my hats to minimize the error. My goal is to make the high-power learners fit the high-difficulty phenomena and make the low-power learners max out their contribution. I don't want gbm's approximating what can be done well using a simple linear model. $\endgroup$ – EngrStudent Jan 3 '18 at 19:23
  • $\begingroup$ -> you seem to be confounding Boosting methods with RandomForest? RandomForest is not a Boosting method. $\endgroup$ – Lucas Roberts Jan 3 '18 at 22:33
  • $\begingroup$ I am not confounding anything. They are both cart models. I treat the boosted (series) CART as an analog to the covariance matrix in the same way that I treat the bagging (parallel) CART as an analog to the mean. That was, however, an "extra" which I can remove if it causes confusion. $\endgroup$ – EngrStudent Jan 4 '18 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.