18
$\begingroup$

Random forests are used for regression. However, from what I understand, they assign an average target value at each leaf. Since there are only limited leaves in each tree, there are only specific values that the target can attain from our regression model. Thus is it not just a 'discrete' regression (like a step function) and not like linear regression which is 'continuous'?

Am I understanding this correctly? If yes, what advantage does random forest offer in regression?

$\endgroup$
23
$\begingroup$

This is correct - random forests discretize continuous variables since they are based on decision trees, which function through recursive binary partitioning. But with sufficient data and sufficient splits, a step function with many small steps can approximate a smooth function. So this need not be a problem. If you really want to capture a smooth response by a single predictor, you calculate the partial effect of any particular variable and fit a smooth function to it (this does not affect the model itself, which will retain this stepwise character).

Random forests offer quite a few advantages over standard regression techniques for some applications. To mention just three:

  1. They allow the use of arbitrarily many predictors (more predictors than data points is possible)
  2. They can approximate complex nonlinear shapes without a priori specification
  3. They can capture complex interactions between predictions without a priori specification.

As for whether it is a 'true' regression, this is somewhat semantic. After all, piecewise regression is regression too, but is also not smooth. As is any regression with a categorical predictor, as pointed out in the comments below.

$\endgroup$
  • 7
    $\begingroup$ Also, regression with only categorical features also wouldn't be smooth. $\endgroup$ – Tim May 14 at 12:41
  • 3
    $\begingroup$ Could a regression with even one categorical feature be smooth? $\endgroup$ – Dave May 14 at 19:59
4
$\begingroup$

It is discrete, but then any output in the form of a floating point number with fixed number of bits will be discrete. If a tree has 100 leaves, then it can give 100 different numbers. If you have 100 different trees with 100 leaves each, then your random forest can theoretically have 100^100 different values, which can give 200 (decimal) digits of precision, or ~600 bits. Of course, there is going to be some overlap, so you're not actually going to see 100^100 different values. The distribution tends to get more discrete the more you get to the extremes; each tree is going to have some minimum leaf (a leaf that gives an output that's less than or equal to all the other leaves), and once you get the minimum leaf from each tree, you can't get any lower. So there's going to be some minimum overall value for the forest, and as you deviate from that value, you're going to start out with all but a few trees being at their minimum leaf, making small deviations from the minimum value increase in discrete jumps. But decreased reliability at the extremes is a property of regressions in general, not just random forests.

$\endgroup$
  • $\begingroup$ The leaves can store any value from the training data (so with the right training data, 100 trees of 100 leaves can store up to 10,000 distinct values). But the returned value is the mean of the chosen leaf from each tree. So the number of bits of precision of that value is the same whether you have 2 trees or 100 trees. $\endgroup$ – Darren Cook May 17 at 7:01
3
$\begingroup$

The answer will depend on what is your definition of regression, see Definition and delimitation of regression model. But a usual definition (or part of a definition) is that regression models conditional expectation. And a regression tree can indeed be seen as an estimator of conditional expectation.

In the leaf nodes you predict the average of the sample observations reaching that leaf, and an arithmetical mean is an estimator of an expectation. The branching pattern in the tree represents the conditioning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.