0
$\begingroup$

In the beginning of https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf, it is explained that:

The common element in all of these procedures is that for the kth tree, a
random vector Θk is generated, independent of the past random vectors
Θ1, ... ,Θk−1 but with the same distribution; 

One way that I understand random forests is, you have some data say of dimensions $N x P$, so now you grow some $Q$ amount of decisions trees which will be used as classifiers, and each tree will get a random subset of the data. However, in the paper there are two inputs to the trees, the data $x$ and a random vector. What is this random vector? Is it just a column array of randomly generated numbers, that adheres to the constraints of independence from the other random vectors ?

$\endgroup$

1 Answer 1

1
$\begingroup$

Breiman says the following:

The common element in all of these procedures is that for the kth tree, a random vector Θk is generated, independent of the past random vectors Θ1, ... ,Θk−1 but with the same distribution; and a tree is grown using the training set and Θk , resulting in a classifier h(x,Θk ) where x is an input vector.

In all the algorithms Breiman is talking about, every time we fit a single tree, we first make some random decisions about how that tree will be fit

  • In bagging, we decide to fit using only some randomly chosen data points.
  • In the random split selection example, we, at each split, randomly choose one of the top $k$ candidate splits.

Breiman introduces the notation $\Theta_k$ for all the random choices we make when fitting the k'th tree. This makes the tree a function of both the data and the random choices, hence the notation

$$h(x, \Theta_k) $$

Random forest goes like

  • For each tree, we first decide to fit using only some randomly chosen data points, then at each split, we choose to only consider a random selection of possible variables.

What you describe here

you have some data say of dimensions NxP, so now you grow some Q amount of decisions trees which will be used as classifiers, and each tree will get a random subset of the data.

is called bagging. Random forest combines bagging with withholding random predictors from tree splits.

$\endgroup$
1
  • $\begingroup$ Hi @MatthewDrury, hope you're doing well. If you don't mind, and if I understood well: 1) Does $\Theta_k$ here stand for the bootstrapping plus the random selection of variables/features (or columns) that are fed to each tree in the random forest algorithm? 2) Is $x$ here the entirety of the sample (an $n\times m$ matrix, where $n$ is number of rows and $m$ is the number of columns) or just the bootstrapped sample fed to each tree? Thanks in advance. $\endgroup$ Commented Apr 30, 2020 at 5:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.