# Can Random Forest Methodology be Applied to Linear Regressions?

Random Forests work by creating an ensemble of decision trees where each tree is created using a bootstrap sample of the original training data (sample of both input variables and observations).

Can a similar process be applied for linear regression? Create k linear regression models using a random bootstrap sample for each of the k regressions

What are the reasons NOT to create a "random regression" like model?

Thanks. If there's something I'm just fundamentally misunderstanding then please let me know.

-
When bootstrap aggregating trees, the overall regression function becomes more and more complex with every tree one adds. On the other hand, when bootstrap aggregating linear functions of the form a_0 + a_1 * x_1 + ... + a_d * x_d , the resulting averaged linear function (after bootstrap aggregating) still has the same linear functional form as the one you start with (i.e. the 'base learner'). –  Andre Holzner Jan 16 at 19:47

I partially disagree with the present answers because the methodology random forest is built upon introduces variance (CARTs built on bootstrapped samples + random subspace method) to make them independent. Once you have orthogonal trees then the average of their predictions tends (in many cases) to be better than the prediction of the average tree (because of the Jensen's inequality). Although CARTs have noticeable perks when subject to this treatment this methodology definitely applies to any model and linear models are no exception. Here is an R package which is exactly what you are looking for. It presents a nice tutorial on how to tune and interpret them and bibliography on the subject: Random Generalized Linear Models.

-

I suppose that, if you were going to do this with $k$ going to infinity, you would obtain the linear regression model you obtain by doing ordinary linear regression with the full sample. Just notice that the average of $k$ structurally equal linear models is again a structurally equal linear model, simply with the parameters averaged (use distributive law). But I didn't do the math and I'm not completely sure.

And here is why it isn't as attractive to do the "random"-thing with linear models as it is with decision trees:

A large decision tree created from a large sample is very likely to overfit the data, and the random forest method fights this effect by relying on a vote of many small trees.

Linear regression on the other hand, is a model that is not very prone to overfitting and thus isn't hurt by training it on the complete sample in the beginning. And even if you have many regressor variables, you can apply other techniques, such as regularization, to combat overfitting.

-