I was watching the following tutorial on Random forests and it says that, "at each node, choose random subset of m features and only consider splitting on those features" I do not understand why we choose m features. Why don't we consider all features?
e.g. Where the features are height, weight, age and m was 2, the random subspace could be height and weight? How does this work?
Thank you in advance!
Just as fitting each tree in the forest with a random subset sample (a bootstrap sample) of the available data adds stochasticity to improve the out-of-sample fit (by reducing the variance component of the error), so does selecting $m$ variables within which to search for each fit add stochasticity. The extra stochasticity makes each tree a poorer model than it would be without the random selection of variables and this is how the Random Forest achieves lower error over a single tree or bagging, which is Random Forests without the random selection of $m$ variables.
Actually, an earlier paper (predating the "random forest" terminology) from Leo Breiman considered all features as you suggest [Breiman 1996]. Below are some relevant excerpts from this article. First, some notation:
A learning set of $\mathcal{L}$ consists of data $\left\{ (y_n,x_n),n=1,...,N\right\}$ where the $y$'s are either class labels or a numerical response. Assume we have a procedure for using this learning set to form a predictor $\varphi(\textbf{x},\mathcal{L})$ -- if the input is $\textbf{x}$ we predict $y$ by $\varphi(\textbf{x},\mathcal{L})$
The idea behind bagging (and, by extension, random forests) is to stochastically construct many different predictors $\varphi$ and aggregate this population predictors into a single predictor, denoted $\varphi_B$, which generally has superior performance to any of the individual predictors. Stochasticity is added by constructing each predictor on a bootstrap sample of the original training set, rather than the entirety of $\mathcal{L}$. As Gavin Simpson mentioned, selecting a random subset of features adds even more stochasticity to the construction of individual predictors. Breiman (1996) writes:
A critical factor in whether bagging will improve accuracy is the stability of the procedure for constructing $\varphi$. If changes in $\mathcal{L}$, i.e. a [bootstrap] replicate of $\mathcal{L}$, produces small changes in $\varphi$, then $\varphi_B$ will be close to $\varphi$. Improvement will occur for unstable procedures where a small change in $\mathcal{L}$ can result in large changes in $\varphi$ . . . The evidence, both experimental and theoretical, is that bagging can push a good but unstable procedures a significant step towards optimality. On the other hand, it can slightly degrade the performance of stable procedures. [Emphasis Added]
The paper then goes on to provide said theoretical and experimental evidence. Breiman's seminal description of random forests (Breiman, 1999) credits Tim Kam Ho for developing "the random subspace method" which adds additional instability by selecting $m$ random features to grow each tree (rather than each node). The original publication (Ho, 1998) appears to be behind a paywall unfortunately.
One final note -- one interesting question is what to choose for $m$? That is, how many random features should we select to split on for each node? There is a section in Breiman (1999) devoted to this question. The basic conclusion is that the choice does not matter substantially, as long as $m$ is relatively small compared to the total number of features.
