Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Random forests are used for regression. However, from what I understand, they assign an average target value at each leaf. Since there are only limited leaves in each tree, there are only specific values that the target can attain from our regression model. Thus is it not just a 'discrete' regression (like a step function) and not like linear regression which is 'continuous'?
Am I understanding this correctly? If yes, what advantage does random forest offer in regression?
This is correct - random forests discretize continuous variables since they are based on decision trees, which function through recursive binary partitioning. But with sufficient data and sufficient splits, a step function with many small steps can approximate a smooth function. So this need not be a problem. If you really want to capture a smooth response by a single predictor, you calculate the partial effect of any particular variable and fit a smooth function to it (this does not affect the model itself, which will retain this stepwise character).
Random forests offer quite a few advantages over standard regression techniques for some applications. To mention just three:
As for whether it is a 'true' regression, this is somewhat semantic. After all, piecewise regression is regression too, but is also not smooth. As is any regression with a categorical predictor, as pointed out in the comments below.
It is discrete, but then any output in the form of a floating point number with fixed number of bits will be discrete. If a tree has 100 leaves, then it can give 100 different numbers. If you have 100 different trees with 100 leaves each, then your random forest can theoretically have 100^100 different values, which can give 200 (decimal) digits of precision, or ~600 bits. Of course, there is going to be some overlap, so you're not actually going to see 100^100 different values. The distribution tends to get more discrete the more you get to the extremes; each tree is going to have some minimum leaf (a leaf that gives an output that's less than or equal to all the other leaves), and once you get the minimum leaf from each tree, you can't get any lower. So there's going to be some minimum overall value for the forest, and as you deviate from that value, you're going to start out with all but a few trees being at their minimum leaf, making small deviations from the minimum value increase in discrete jumps. But decreased reliability at the extremes is a property of regressions in general, not just random forests.
The answer will depend on what is your definition of regression, see Definition and delimitation of regression model. But a usual definition (or part of a definition) is that regression models conditional expectation. And a regression tree can indeed be seen as an estimator of conditional expectation.
In the leaf nodes you predict the average of the sample observations reaching that leaf, and an arithmetical mean is an estimator of an expectation. The branching pattern in the tree represents the conditioning.
It's perhaps worth adding that Random Forest models can't extrapolate outside the range of the training data, since their lowest and highest values are always going to be averages of some subset of the training data; there is a nice graphical example here.
