What is the intuition behind why randomForest works well with prediction on lots of applied datasets? No fancy math required for an answer-but if you have some that helps intuition great.
The kind of intuition I like to have is driven by "physics" and connects to "first principles".
You can think of the random forest is a machine, like a flour mill. You put something in, and you get something else out. The two ways to gain intuition are to "look at the output grain" and look at how the machine is put together (assembled).
Going into full depth would take a book. This is a few paragraphs, so it is going to be a relatively shallow exploration.
This is about the a single dust of flour:
If you must bet on a horse race, and you know as little about formal/historic betting as I do, then you might look at the mean times for each of the last several races of all the horses and bet that one that has higher consistent performance than some others is likely to outperform those others. This is a very simple mean. It doesn't account for other factors, but in the long run, it is likely to be a decent predictor of typical performance. You would lose less money than if you uniformly randomly picked a horse and bet on it to win. In this sense, in the long run, the mean can be thought of as the "best" one-parameter estimator of "typical" performance. The "learner" is a single number that doesn't change, but it has some power.
If we instead trained a 2-parameter estimator on each horse, you might have one "constant" parameter and one "time-varying" parameter. This would be a simple linear model.
Now the constant-time estimation of horse race has a number of flaws. Performance changes. Horses get older. Some get sick. There is doping, which is illegal. I would guess that a familiar rider might get more performance than an unfamiliar one.
In the grand scheme of things, accounting for those sources of variation is going to give much better predictive accuracy. The basic mean is going to do that relatively poorly in the grand scheme, but in very small parcels (boxy regions of the input space with volume and coordinates, sometimes also called voxels) it can do fairly well.
And that is a "grain of dust" of a classification and regression tree (CART). Within a box, the model is typically either a mean, or a linear function.
This is about how the model is put together:
Now for the second problem. The right small parcel isn't easy to find: it is easy to get wrong. Typical analytic methods aren't very robust and finding the "edges" where the basic model breaks down. The "entropy" or "variance" methods are related to the square of the derivative, and given noise and sparse sampling, these tend to be easier to estimate wrongly. The parallel "ensemble", using many CART trees with uniformly random different training inputs, then averaging the array of outputs to find the "mean" edge, gives better results. It turns out that in the process of finding the "edge" we also get multiple, slightly different, overlapping models for the "interior function" so we get a better central estimate for a better box.
So in its atomic form, in its soul it is a central tendency or a line. The way the bounds for this "atom" are found, using the parallel ensemble, overcomes the weakness of the second derivative of noise in data, and is another way of thinking of a mean: the mean of bootstrapped models.
(This makes me think a multivariate, nonuniform, Savitzky-Golay estimator might also be useful for finding "splits" for branches in CART models. update: we don't need multivariate, or non-uniform. A fun consequence is that we can split at a partial value that is data-driven using an approach analogous to sub-pixel image registration. Because it is frequency-domain, it should be relatively fast as well. update2: I like how a 5 or 7 element filter can strongly indicate variance. New SO Question.)
There are some places where a random forest is not ideal. link. Personally, I think that when you apply it where it was meant to go, it cannot be beat. I think that it is about central tendency, not high accuracy. I think that it is very fast computationally, and handles sometimes very high dimensional data relatively well. There are no "silver bullets", no "leathermen".
One of the ways I like to treat an RF, is as an extended version of the mean as follows. In many machine learning applications you center and scale your data before you put it into the learner. The idea is for the high-accuracy tool to handle the hard parts, and leave the easy parts to easy tools.