3
$\begingroup$

I'm reading Machine Learning - A First Course for Engineers and Scientists. On page 168 they give a rough explanation for why bagging works. I'm a little confused by their explanation.

They consider an ensemble of $B$ bootstrapped datasets $\mathcal{T}^{(b)}$ with $b=1,2, \ldots, B$. They denote the prediction from the $b$-th bootstrapped dataset $\mathcal{T}^{(b)}$ (taken from the original dataset $\mathcal{T}$) and evaluated at the point $\mathbf{x}_\star$ as $\tilde{y}^{(b)}_\star \equiv y(x_\star; \mathcal{T}^{(b)})$. They assume that the average of these predictions is $$ \mathbb{E}[\tilde{y}^{(b)}_\star] = \mu^2$$ the variance is $$ \mathrm{Var}[\tilde{y}^{(b)}_\star] = \sigma^2$$ and the average correlation between predictions is $$ \text{avg cor}[\tilde{y}^{(b)}_\star] =\frac{1}{B(B-1)}\sum_{b\ne c}\mathbb{E}[(\tilde{y}^{(b)}_\star-\mu)(\tilde{y}^{(c)}_\star-\mu)] = \rho \sigma^2$$ They state that

"All base models, and hence their predictions, originate from the same data $\mathcal{T}$ (via the bootstrap), and $\tilde{y}^{(b)}_\star$ are therefore identically distributed but correlated."

Their explanation goes like this: the average of the bagged prediction is $$ \mathbb{E}[\frac1B \sum^B_{b=1}\tilde{y}^{(b)}_\star] = \sigma^2$$ and the variance is $$ \mathrm{Var}[\frac1B \sum^B_{b=1}\tilde{y}^{(b)}_\star] = \frac{1-\rho}{B} \sigma^2+\rho \sigma^2 $$ If $\rho<1$, then variance is reduced by increasing bootstrap samples.

I'm a bit confused by exactly how they treat $\tilde{y}^{(b)}_\star$ as random variables i.e. what probability distribution are we averaging over in $\mathbb{E}[\tilde{y}^{(b)}_\star]$? Since the point is that the bootstrapping procedure decreases the variance in the bias-variance decomposition of the expected new data error, I'm supposing that the randomness is from draws of the original dataset $\mathcal{T}$. But are we supposing that the indices of bootstrap samples are fixed (i.e. ${\mathcal{T}'}^{(b)}$ is always constructed from taking the datapoints with indices $i \in \lbrace i^{(b)}_1, i^{(b)}_2, \ldots, i^{(b)}_n \rbrace$ but ${\mathcal{T}'}^{(b)}$ is random), or are they also random between draws of $\mathcal{T}$ ?

Also, can anyone give an intuitive explanation of what sort of data and model characteristics would increase $\rho$?

$\endgroup$

1 Answer 1

3
$\begingroup$

It's also random between draws, to ensure lower $\rho$.

In random forests, random subsets of variables (eg $\sqrt{n_{features}}$) are used as candidates to build trees to reduce $\rho$ (ie tree 1 is only allowed to use features 1,5,6, tree 2 is only allowed 4,7,8, etc)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.