10
$\begingroup$

In all the (regression) random forest papers I've read, when it comes the time to gather the predictions of all the trees, we take the average value as the prediction.

My question is why do we do that?

Is there is a statistical justification for taking the average?

EDIT: To clarify the question, I know it's possible to use other aggregation functions (we use the mode for classification), I'm mostly interested in whether there is some theoretical justification behind the choice of the average function.

Improve this question
$\endgroup$
2

6 Answers 6

Reset to default
5
$\begingroup$

I've always thought about the averaging in terms of the bias-variance tradeoff. If I remember correctly Leo Breiman hinted at this in the randomForest paper with his statement "... are more robust with respect to noise."

The explanation goes like this: basically you are taking a bunch of trees that are grown to full length-no pruning-so you know they will each be biased by themselves. However, the random sampling that induces each tree in the forest should induce under-bias as often as over-bias. So by taking an average you then eliminate the bias of each tree-the over+under biases canceling. Hopefully in the process you also reduce the variance in each tree and so the overall variance should be reduced as well.

As indicated by the other answers to the post, this might not be the only reason for averaging.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Accepting this, as the answer seems that the average is chosen out of "intuition" rather a particular theoretical motivation, contrasting to the theoretically motivated answer on GLMs: stats.stackexchange.com/q/174390/16052 $\endgroup$
    – Bar
    Commented Oct 6, 2017 at 14:28
  • 1
    $\begingroup$ @Bar, with decision trees, the problem is really that global optimization is NP-hard so greedy optimization is done. The greedy optimization for each tree doesn't tell us about the forest. Unfortunately, the math for this problem is less developed than either of us would like. $\endgroup$
    – Lucas Roberts
    Commented Oct 6, 2017 at 14:56
6
$\begingroup$

When using the average, you are saying two things:

  1. Outliers are not a huge problem (otherwise you would use the median or at least filter out some outliers before taking the average)
  2. Every prediction has the same weight (otherwise you would factor in weights)

You shouldn't expect there to be huge outliers since you can make the sample size big enough for them to matter less in the average and since you would expect a minimum of stability from the predictions of the individual trees.

There is no reason to think some trees should have more predictive weight than others, nor a way to determine such weights.

You cannot really use mode since the predictions are on a continuous scale. For example, if you had the predictions 80 80 100 101 99 98 97 102 103 104 96, mode would predict as 80. That cannot be what you want. If all values have distinct decimals, mode wouldn't know how to decide.

Other averages than the arithmetic mean exist, like the geometric mean and the harmonic mean. They are designed to pull the average down if there are some low values in the series of data. That's not what you want here either.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ If outliers is a concern, there are alternatives in between the median and the sample mean, like winsorized or trimmed means, which could give good outlier protection while being more efficient than the median. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Oct 5, 2017 at 13:35
3
$\begingroup$

Of course you could use any aggregation function that is useful in your particular situation. The median is a good way of making a small sample robust against outliers. In regression forests you can usually influence the sample size to avoid having the problem of small sample sizes. Thus the mean seems sensible in a very large fraction of use cases.

Improve this answer
$\endgroup$
1
$\begingroup$

Wouldn't it be possible as well to take the median, mode, or some other aggregate function?

Random Forest classification (i.e. not probability estimation) is based on the mode of the predictions (majority voting), so yeah, you can aggregate the results as you like.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the answer, I've added a clarification to my question. I know it's possible to use other aggregation functions, what I'm wondering is if there's any theoretical reason for the choice of average. $\endgroup$
    – Bar
    Commented Oct 5, 2017 at 11:39
1
$\begingroup$

First things first. As many other people said you can use other metrics but the average is the "default" option.

As a default option, one would set a function that works under some mild conditions

Now, If you think about it, a random forest is a collection of trees and each of these trees has the objective to estimate your numeric response variable.

Additionally, as @David Ernst correctly mentions:

There is no reason to think some trees should have more predictive weights as others, nor a way to determine such weights.

Furthermore, there is no reason to think that these trees will have different standard deviations. Again, under mild conditions!

That being said, the average should work because of the Weak law of large numbers

Improve this answer
$\endgroup$
-1
$\begingroup$

In ensemble. Averaging is prioritizing more on confidence rather than majority.

Example you have 3 trees,

2 of them vote A with 22% confidence and 1 voted B with 90% confidence.

If we use majority we get vote A. Average of 22, N, N If we use confidence we get vote B. Average of 90, N, N

It would make sense to go with the 90% confidence since its more sure than the majority of others with only 22% confidence.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It is not quite clear to me what your point is. Can you edit to clarify it perhaps? What would happen if there were 100 A and only one B with the same set of confidence ratings? $\endgroup$
    – mdewey
    Commented Feb 12, 2018 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.