Ensemble vs statistical power In regression modelling, I've seen two schools of thought: ensemble model vs. focus on statistical power (using one model). Proponents of ensemble models (i.e., bagging) argue that:
Suppose $\epsilon_i$ is error of model $i$, there are $k$ models, then error of the average model is (Goodfellow et al., 2016),
$E[(\frac{1}{k}\sum_{i}\epsilon_i)^2] = \frac{1}{k^2}E[\sum_i(\epsilon_i^2+\sum_{j \ne i}\epsilon_i\epsilon_j)] = \frac{1}{k}v + \frac{k-1}{k}c,$
where $v = E[\epsilon_i^2]$ is model error and $c = E[\epsilon_i\epsilon_j]$ is model covariance. So it is beneficial to build many different estimators and average over them. One way to build very different estimators is by using bootstrapping, so training data for each model is different.
The other camp argues that the more data you have in a regression, the higher the power of the model and thus the more certain you are in that the observed effect (the regression coefficient) is not random/false positive. Thus, you should throw in as much data as possible into one regression. Say you have 10 million observations and you're very certain of the effect. Why do you need an ensemble model?
My understanding is that this is sort of answering two different questions. The ensemble model is focused on predictive power and not so much on which covariates explained the data, whereas the latter is focused on explanatory power and out of sample prediction performance is heavily reliant on out of sample data being drawn from the same distribution as training data.
Which one is correct and how would you argue for one or the other?
 A: Well bagging a linear model will just converge in prediction to NOT bagging depending on your sample rate and number of estimators.  This is not true if you use some piecewise functions in your model though.  But this is just because bagging won't add any complexity to a linear model so you just bias the model by some amount then another model and so on, but the average just becomes essentially un-biased.  Now you may want to use bootstrapping in general to do statistics on your coefficients/error rather than relying on the standard equations to derive those variances but once again I think those numbers should all be close with enough samples and with 10 million samples I wouldn't be worried about whether to bag or not, just do the normal stuff.
Now when do we typically want an ensemble approach?  Typically for prediction when adding two models also adds complexity such as a decision tree or some piecewise function in our regression. OR if you use a different ensembling technique, gradient boosting can actually add regularization to your coefficients similar in effect to a ridge regression, once again though this makes interpreting our coefficients fuzzy so typically done for predictive power.
I think it's important to think of the bias-variance tradeoff.  Ensembling is done to increase bias in low bias models in order to decrease their variance to new data.  But typically a linear model is not 'low' enough bias for ensembling to do much so we have to decrease the bias (and increase the variance) with polynomial expansions or something like that first, then we can look to ensembling.
