I'm reading Machine Learning - A First Course for Engineers and Scientists. On page 168 they give a rough explanation for why bagging works. I'm a little confused by their explanation.
They consider an ensemble of $B$ bootstrapped datasets $\mathcal{T}^{(b)}$ with $b=1,2, \ldots, B$. They denote the prediction from the $b$-th bootstrapped dataset $\mathcal{T}^{(b)}$ (taken from the original dataset $\mathcal{T}$) and evaluated at the point $\mathbf{x}_\star$ as $\tilde{y}^{(b)}_\star \equiv y(x_\star; \mathcal{T}^{(b)})$. They assume that the average of these predictions is $$ \mathbb{E}[\tilde{y}^{(b)}_\star] = \mu^2$$ the variance is $$ \mathrm{Var}[\tilde{y}^{(b)}_\star] = \sigma^2$$ and the average correlation between predictions is $$ \text{avg cor}[\tilde{y}^{(b)}_\star] =\frac{1}{B(B-1)}\sum_{b\ne c}\mathbb{E}[(\tilde{y}^{(b)}_\star-\mu)(\tilde{y}^{(c)}_\star-\mu)] = \rho \sigma^2$$ They state that
"All base models, and hence their predictions, originate from the same data $\mathcal{T}$ (via the bootstrap), and $\tilde{y}^{(b)}_\star$ are therefore identically distributed but correlated."
Their explanation goes like this: the average of the bagged prediction is $$ \mathbb{E}[\frac1B \sum^B_{b=1}\tilde{y}^{(b)}_\star] = \sigma^2$$ and the variance is $$ \mathrm{Var}[\frac1B \sum^B_{b=1}\tilde{y}^{(b)}_\star] = \frac{1-\rho}{B} \sigma^2+\rho \sigma^2 $$ If $\rho<1$, then variance is reduced by increasing bootstrap samples.
I'm a bit confused by exactly how they treat $\tilde{y}^{(b)}_\star$ as random variables i.e. what probability distribution are we averaging over in $\mathbb{E}[\tilde{y}^{(b)}_\star]$? Since the point is that the bootstrapping procedure decreases the variance in the bias-variance decomposition of the expected new data error, I'm supposing that the randomness is from draws of the original dataset $\mathcal{T}$. But are we supposing that the indices of bootstrap samples are fixed (i.e. ${\mathcal{T}'}^{(b)}$ is always constructed from taking the datapoints with indices $i \in \lbrace i^{(b)}_1, i^{(b)}_2, \ldots, i^{(b)}_n \rbrace$ but ${\mathcal{T}'}^{(b)}$ is random), or are they also random between draws of $\mathcal{T}$ ?
Also, can anyone give an intuitive explanation of what sort of data and model characteristics would increase $\rho$?
