Can Random Forest Methodology be Applied to Linear Regressions?

Question

Random Forests work by creating an ensemble of decision trees where each tree is created using a bootstrap sample of the original training data (sample of both input variables and observations).

Can a similar process be applied for linear regression? Create k linear regression models using a random bootstrap sample for each of the k regressions

What are the reasons NOT to create a "random regression" like model?

Thanks. If there's something I'm just fundamentally misunderstanding then please let me know.

Answer 1

I partially disagree with the present answers because the methodology random forest is built upon introduces variance (CARTs built on bootstrapped samples + random subspace method) to make them independent. Once you have orthogonal trees then the average of their predictions tends (in many cases) to be better than the prediction of the average tree (because of Jensen's inequality). Although CARTs have noticeable perks when subject to this treatment this methodology definitely applies to any model and linear models are no exception. Here is an R package which is exactly what you are looking for. It presents a nice tutorial on how to tune and interpret them and bibliography on the subject: Random Generalized Linear Models.

Answer 2

To put @ziggystar's response in terms of machine learning jargon: the idea behind bootstrap aggregation techniques (e.g. Random Forests) is to fit many low-bias, high-variance models to data with some element of "randomness" or "instability." In the case of random forests, instability is added through bootstrapping and by picking a random set of features to split each node of the tree. Averaging across these noisy, but low-bias, trees alleviates the high variance of any individual tree.

While regression/classification trees are "low-bias, high-variance" models, linear regression models are typically the opposite - "high-bias, low-variance." Thus, the problem one often faces with linear models is reducing bias, not reducing variance. Bootstrap aggregation is simply not made to do this.

An addition problem is that bootstrapping may not provide enough "randomness" or "instability" in a typical linear model. I would expect a regression tree to be more sensitive to the randomness of bootstrap samples, since each leaf typically only holds a handful of data points. Additionally, regression trees can be stochastically grown by splitting the tree on a random subset of variables at each node. See this previous question for why this is important: Why are Random Forests splitted based on m random features?

All that being said, you can certainly use bootstrapping on linear models [LINK], and this can be very helpful in certain contexts. However, the motivation is much different from bootstrap aggregation techniques.

Answer 3

I suppose that, if you were going to do this with $k$ going to infinity, you would obtain the linear regression model you obtain by doing ordinary linear regression with the full sample. Just notice that the average of $k$ structurally equal linear models is again a structurally equal linear model, simply with the parameters averaged (use distributive law). But I didn't do the math and I'm not completely sure.

And here is why it isn't as attractive to do the "random"-thing with linear models as it is with decision trees:

A large decision tree created from a large sample is very likely to overfit the data, and the random forest method fights this effect by relying on a vote of many small trees.

Linear regression on the other hand, is a model that is not very prone to overfitting and thus isn't hurt by training it on the complete sample in the beginning. And even if you have many regressor variables, you can apply other techniques, such as regularization, to combat overfitting.

Answer 4

I agree with @ziggystar. As the number of bootstrap samples $k$ converges to infinity, bagged estimate of the linear model converges to the OLS (Ordinary Least Squares) estimate of the linear model run on the whole sample. The way to prove this is seeing that bootstrap "pretends" that the population distribution is the same as the empirical distribution. As you sample more and more data sets from this empirical distribution, the average of estimated hyperplanes will converge to the "true hyperplane" (which is OLS estimate run on the whole data) by asymptotic properties of Ordinary Least Squares.

Also, bagging is not always a good thing. Not only does it not fight the bias, it may increase the bias in some peculiar cases. Example: $$ X_1, X_2, ..., X_n \sim Be(p) $$ (Bernoulli trials which take value 1 with probability $p$ and value 0 with probability $1-p$). Further, let us define parameter $$ \theta = 1_{\{p > 0\}} $$ and try to estimate it. Naturally, it suffices to see a single data point $X_i = 1$ to know that $\theta = 1$. The whole sample may contain such a data point and allow us to estimate $\theta$ without any error. On the other hand, any bootstrap sample may not contain such a data point and lead us to wrongly estimating $\theta$ with 0 (we adopt no Bayesian framework here, juts good old method of maximum likelihood). In other words, $$ {\rm Bias}_{\rm\ bagging} = {\rm Prob(in\ a\ bootstrap\ sample\ X_{(1)} = ... = X_{(n)} = 0)} > 0, $$ conditional on $\theta = 1$.