7
$\begingroup$

There are two methods to select subset of features during a tree construction in random forest:

According to Breiman, Leo in "Random Forests":

“… random forest with random features is formed by selecting at random, at each node, a small group of input variables to split on.”

Tin Kam Ho used the “random subspace method” where each tree got a random subset of features.

I can imagine that by selecting a subset of features at each node is more superior as the correlated variables can still be involved in the whole tree construction. Whereas if we select a subset of features for each tree, one of the correlated variables will lose its importance.

Are there any other reasons why one method can perform better than the other one?

$\endgroup$
0
$\begingroup$

In context of tidy data, one bootstraps on samples(rows) and one bootstraps on both samples(rows) and variables(columns). They, as far as I know, always bootstrap in rows.

Here are the rules for "tidy" originally put forth by Hadley Wickham [1,2]:

  1. Each variable forms a column.
  2. Each observation forms a row.
  3. Each type of observational units forms a table.

So the question becomes "what is the advantage of bootstrapping on columns".

It gives you what bootstrapping always gives, but applied to the column space:

  • robust characterization. when a column is important, and excluded, error is much larger and vis versa. This can add emphasis on giving higher weight to higher importance variables, and given that tree-weights are inverse to error, this can reduce the impact of less important variables.
  • Accelerated compute: when you operate on less data, ceteris paribus, your algo runs faster. If you make each tree with 75% of the columns, then they construct faster.
$\endgroup$
  • $\begingroup$ The question is about the effect of tree-wise versus node-wise column sampling. $\endgroup$ – Michael M Jan 15 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.