0
$\begingroup$

OOB error rate in the documentation is defined as

(classification only) vector error rates of the prediction on the input data, the i-th element being the (OOB) error rate for all trees up to the i-th. https://cran.r-project.org/web/packages/randomForest/randomForest.pdf

I need help in understanding this. I have 500 trees, and generally the error rate decreases as the number of trees increases. Why would this decrease as the number of trees increases?

When I print model$err.rate[,1], and we see the 500 oob error values, is this specifically for the last prediction, or is it some average?

When you print(Forest) and it returns the OOB estimate of error rate, how does this single value relate to the 500 values printed previously, and how does the confusion matrix relate?

$\endgroup$
1

1 Answer 1

1
$\begingroup$

Why would this decrease as the number of trees increases?

From Philipp Probst & Anne-Laure Boulesteix "To tune or not to tune the number of trees in random forest?" we find an answer.

The expected error rate can be a non-monotonous function of the number of trees. The expected error rate (equiv. $\text{error rate} = 1 - \text{accuracy}$) as a function of $T$ the number of trees is given by $$ E(e_i(T)) = P\left(\sum_{t=1}^T e_{it} > 0.5\cdot T\right) $$ where $e_{it}$ is a binomial r.v. with expectation $E(e_{it}) = \epsilon_i$, the decision of a particular tree indexed by $t$. This function is increasing in $T$ for $\epsilon_{i} > 0.5$ and decreasing in $T$ for $\epsilon_{i} < 0.5$. The authors observe

We see that the convergence rate of the error rate curve is only dependent on the distribution of the $\epsilon_i$ of the observations. Hence, the convergence rate of the error rate curve is not directly dependent on the number of observations $n$ or the number of features, but these characteristics could influence the empirical distribution of the $\epsilon_i$’s and hence possibly the convergence rate as outlined in Section 4.3.1

So, to directly answer your question, the decrease that you observe is related to the idiosyncratic component of the data. The decision of any tree is a certain kind of random variable, and the expected error rate for any number of trees $T$ is solely a feature of the $\epsilon_i$ which will be different for each data set. You may find that increasing the number of trees causes the error rate to oscillate a bit, perhaps because some datum $j$ has $\epsilon_j$ very near to $\frac{1}{2}$, so it's flipping between being classified one way and another way.

Instead of tracking error rate or AUC as a function of $T$, I suggest tracking log-loss or Brier score as a function of $T$, because these measurements are monotonically decreasing in expectation as a function of $T$. See: Do we have to tune the number of trees in a random forest?

When I print model$err.rate[,1], and we see the 500 oob error values, is this specifically for the last prediction, or is it some average?

This is the OOB rate for all trees with index less than or equal to the row index. That's what your quote from the documentation means when it says "the i-th element being the (OOB) error rate for all trees up to the i-th." Every time you add a tree, this could change the cumulative OOB error rate, so this is a convenient way to track how that change happens as you add trees.

When you print(Forest) and it returns the OOB estimate of error rate, how does this single value relate to the 500 values printed previously, and how does the confusion matrix relate?

The OOB rate printed by print(Forest) is the OOB rate for the entire ensemble. If you have T trees in Forest, it's Forest$err.rate[T,1] expressed as a percent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.