1
$\begingroup$

I am trying to code my own, simple version of RandomForest function in R for learning purposes. However I have a hard time understanding the concept of the out-of-bag error.

Is it simply done by computing for each tree the error on sample not seen during training? So for example if my tree was build of 60% of dataset, I compute the error on remaining 40% of dataset, I repeat that logic for each tree and average the results?

Or am I misunderstanding something?

[Edit] My confusion comes from this definition:

enter image description here

Because this definition seems different to what my intuition says.

$\endgroup$
  • 1
    $\begingroup$ Wouldn't the remaining sample be 40%? $\endgroup$ – StatsStudent Jan 19 at 17:38
  • $\begingroup$ But other than that your thinking is right $\endgroup$ – peteR Jan 19 at 17:39
  • $\begingroup$ BTW, this question may have already been answered on SO: stackoverflow.com/questions/18541923/… $\endgroup$ – StatsStudent Jan 19 at 17:42
1
$\begingroup$

You train each tree, $T$, with a subset of the training set (a bootstrap sample), and some of the samples are left out. So, for each tree in the forest, there are out of the bag samples. If I didn't misunderstand you, you test these tree-specific OOB samples on the tree, $T$, and calculate the error, do this for every tree, and average the errors, which is not the way you do normally, and in the explanation in your image. Here, you test your samples on trees, not the forest.

Just like you have a subset of samples for each tree, conversely, for each sample, $x_i$, you have a set of trees that your sample is out of bag, e.g. $T_i=\{T_{i1},T_{i2},...,T_{im}\}$ set of trees that don't contain $x_i$ in their sample subset. You calculate predictions of $x_i$ on this smaller forest, average them, which creates $\hat{y}^{oob}$ and then calculate your error. You do this for each sample in your dataset, and obtain the OOB error.

$\endgroup$
  • $\begingroup$ thank you ! but I have one further question. Let's say that my explanatory variables are denoted Xi = [Xi1,Xi2..Xim]. In each tree I randomly sample 60% of m variables. If so, every tree will be 'out of sample' as I will never have a tree that used (xi,yi) -> all variables in row 'i' as a train sample. I hope that my question makes sense. $\endgroup$ – user208493 Jan 21 at 19:06
  • $\begingroup$ But, you're using that specific sample, i.e. $x_i,y_i$; it doesn't matter if you exploit all of its features or not. Testing philosophy is based on unseen samples. $\endgroup$ – gunes Jan 21 at 19:51
  • $\begingroup$ Ok thank you ! and what if some xi,y weren't used in any tree? how to compute the OOB classifier for them? $\endgroup$ – user208493 Jan 21 at 20:00
  • $\begingroup$ If the sample is not used in any trees, then every tree will include it in their test, (OOB) set for OOB error calculation, i.e. $T_i$ is the set of all trees. $\endgroup$ – gunes Jan 21 at 20:02
  • $\begingroup$ oh sorry, the question was supposed to be the other way around, what if some xi,yi were used in EVERY tree, how to compute OOB fo them? $\endgroup$ – user208493 Jan 21 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.