# Can any Models be "Bagged"?

I have been learning about "bagging" (bootstrap aggregation) - supposedly, there are many types of statistical models can be bagged together. For example, CART Decision Trees can be "bagged" together to from a Random Forest.

I am guessing that statistical models that are bagged together often lose some of their interpretability. For example, if you were to "bag" several regression models together, I am not sure if the regression coefficients of an ensemble of "bagged" regression models have the same interpretability and explanatory power as a single regression model.

This brings me to my question: Suppose some researchers are interested in fitting a Multi State Markov Model (e.g. https://cran.r-project.org/web/packages/msm/vignettes/msm-manual.pdf) to a very large dataset - suppose this dataset is so large (millions and millions of rows) that a Multi State Model can not be fit to the entire dataset. Thus, the researchers decide to take multiple random samples from this dataset, fit individual Multi State Models to each of these random samples - could these models then somehow be "bagged" together (I am not sure how transition probabilities from different Markov Chains can be aggregated together) and still keep their interpretability? Or is "bagging" only used for very specific models and instances (e.g. Random Forest)?

Thanks!

You ask "Or is 'bagging' only used for very specific models and instances (e.g. Random Forest)?"

If you truly want to, you can bag any model. The reason that bagging is almost synonymous with random forest is that bagging is more effective for high-variance models like decision trees. Again, Elements of Statistical Learning (p. 589) writes

Not all estimators can be improved by shaking up the data like this [via bagging]. It seems that highly nonlinear estimators, such as trees, benefit the most. For bootstrapped trees, $$\rho$$ [the correlation of estimators] is typically small (0.05 or lower is typical; see Figure 15.9), while $$\sigma^2$$ [their variance] is not much larger than the variance for the original tree. On the other hand, bagging does not change linear estimates, such as the sample mean (hence its variance either); the pairwise correlation between bootstrapped means is about 50% (Exercise 15.4).

This is made more precise in the surrounding text, but this paragraph is a good summary.

Whether a model is "interpretable" is up to you to decide. Elements of Statistical Learning (p. 286) provides some commentary:

Note that when we bag a model, any simple structure in the model is lost. As an example, a bagged tree is no longer a tree. For interpretation of the model this is clearly a drawback. More stable procedures like nearest neighbors are typically not affected much by bagging. Unfortunately, the unstable models most helped by bagging are unstable because of the emphasis on interpretability, and this is lost in the bagging process.

Regarding memory-constrained computation of Markov Models, I have 2 comments.

1. The bootstrap sample has a similar memory cost as the original sample. In the naive implementation, you're drawing a random sample equal to the original data size. If you use weighting instead of naively gathering duplicates of the data, this might economize the memory consumption a little, since on average the bootstrap sample includes $$1 - \exp(-1) \approx 63\%$$ of the original data, but this seems marginal on the whole.

2. You can just fit the model out-of-memory. In the past when I've faced this problem, I've used a neural network library, and hand-coded the model. In the NN forward pass, I implemented Baum-Welch in the NN library. Then the backward pass is automatically done via backprop. In this way, we can update the model incrementally using whatever mini-batch size fits in memory.

Bagging is usually used for model so called the "weak learner" like Decision Tree. The reason behind is those weak learner usually overfit and lack of generalization power (High Variance).

With the application of Bagging , researcher found that it can reduce the variance of model (prevent overfitting). Althought some suggest that bagging would sometime reduce bias, mostly people use bagging to reduce overfitting. For underfitting, boosting may be a better idea.

For your case, your MSM model seems is underfitting the data instead of overfitting. Bagging might not be a good idea.