1
$\begingroup$

Usually if we have $n$ observations, for each tree with form a bootstrapped subsample of size $n$ with replacement. On googling it one common explanation I've seen is that with replacement sampling is necessary for independence of individual trees.

But why can't we just resample as follows: for tree 1, randomly sample $m$ observations without replacement out of the $n$, where $m$ is still large enough (of course, provided that $n$ is large enough in the first place). Then replenish all observations and repeat the resampling for tree 2, and so on.

Even in this case, I'd imagine that the individual subsamples would be independent. So is there an additional reason for resampling with replacement in bagging?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Independent is the wrong word, but yes, this is sometimes done like this.

Sampling without replacement is supported by many implementations of random forests.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer! What would be the correct term instead of independent? I thought correlated but that implies a linear relationship, so I think "correlated" would also be a wrong term $\endgroup$
    – user9343456
    May 23, 2021 at 15:58
  • $\begingroup$ The tree predictions of a random forest are extremely strongly correlated across trees. Still, they are not perfectly correlated, so that the variance of the ensemble is still lower than the one from a single tree. $\endgroup$
    – Michael M
    May 23, 2021 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.