Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Random Forests work by creating an ensemble of decision trees where each tree is created using a bootstrap sample of the original training data (sample of both input variables and observations).

Can a similar process be applied for linear regression? Create k linear regression models using a random bootstrap sample for each of the k regressions

What are the reasons NOT to create a "random regression" like model?

Thanks. If there's something I'm just fundamentally misunderstanding then please let me know.

share|improve this question
When bootstrap aggregating trees, the overall regression function becomes more and more complex with every tree one adds. On the other hand, when bootstrap aggregating linear functions of the form a_0 + a_1 * x_1 + ... + a_d * x_d , the resulting averaged linear function (after bootstrap aggregating) still has the same linear functional form as the one you start with (i.e. the 'base learner'). – Andre Holzner Jan 16 '14 at 19:47
@Andre Holzner - what you say it true, but, but, but... doing this random forrest is actually a form of regularisation, in a similar class to ridging. I'll tell you a secret, a regression tree is actually a linear model - similar class to splines. putting my Bayesian hat on, the random forrest regulariser would likely correspond roughly to the "spike and slab" priors used in Bayesian context. – probabilityislogic Apr 29 '14 at 11:28
@probabilityislogic, can you explain? – Simon Kuang May 28 '15 at 3:58
You can think of trees as the linear model $ y=Z_t\theta_t+e $. $Z_t$ is a design matrix indicating which terminal node each observation belongs to for tree $ t $, and $\theta_t $ is the corresponding vector of terminal node predictions. Any tree can be described in this way - choosing a tree is equivalent to standard linear model selectionin the space of $ Z_t $ - of which there is $2^n $ possible "terminal node" cconfigurations I think (where $ n $ is the training sample size). – probabilityislogic May 28 '15 at 11:33
up vote 3 down vote accepted

I partially disagree with the present answers because the methodology random forest is built upon introduces variance (CARTs built on bootstrapped samples + random subspace method) to make them independent. Once you have orthogonal trees then the average of their predictions tends (in many cases) to be better than the prediction of the average tree (because of the Jensen's inequality). Although CARTs have noticeable perks when subject to this treatment this methodology definitely applies to any model and linear models are no exception. Here is an R package which is exactly what you are looking for. It presents a nice tutorial on how to tune and interpret them and bibliography on the subject: Random Generalized Linear Models.

share|improve this answer

To put @ziggystar's response in terms of machine learning jargon: the idea behind bootstrap aggregation techniques (e.g. Random Forests) is to fit many low-bias, high-variance models to data with some element of "randomness" or "instability." In the case of random forests, instability is added through bootstrapping and by picking a random set of features to split each node of the tree. Averaging across these noisy, but low-bias, trees alleviates the high variance of any individual tree.

While regression/classification trees are "low-bias, high-variance" models, linear regression models are typically the opposite - "high-bias, low-variance." Thus, the problem one often faces with linear models is reducing bias, not reducing variance. Bootstrap aggregation is simply not made to do this.

An addition problem is that bootstrapping may not provide enough "randomness" or "instability" in a typical linear model. I would expect a regression tree to be more sensitive to the randomness of bootstrap samples, since each leaf typically only holds a handful of data points. Additionally, regression trees can be stochastically grown by splitting the tree on a random subset of variables at each node. See this previous question for why this is important: Why are Random Forests splitted based on m random features?

All that being said, you can certainly use bootstrapping on linear models [LINK], and this can be very helpful in certain contexts. However, the motivation is much different from bootstrap aggregation techniques.

share|improve this answer
Thanks for the links and response. If the randomness method is useful for "low bias, high variance" models, are there any methodologies for dealing with the opposite type of models "high bias, low variance" ? – Rick Jan 16 '14 at 20:37
If you have a low bias, high variance model, methodologies like bagging can reduce the variance at a slight increase in bias. If you have high bias, low variance, use a model that's lower bias and higher variance - like a polynomial regression or more generally kernel methods. – Joe Jan 16 '14 at 23:57

I suppose that, if you were going to do this with $k$ going to infinity, you would obtain the linear regression model you obtain by doing ordinary linear regression with the full sample. Just notice that the average of $k$ structurally equal linear models is again a structurally equal linear model, simply with the parameters averaged (use distributive law). But I didn't do the math and I'm not completely sure.

And here is why it isn't as attractive to do the "random"-thing with linear models as it is with decision trees:

A large decision tree created from a large sample is very likely to overfit the data, and the random forest method fights this effect by relying on a vote of many small trees.

Linear regression on the other hand, is a model that is not very prone to overfitting and thus isn't hurt by training it on the complete sample in the beginning. And even if you have many regressor variables, you can apply other techniques, such as regularization, to combat overfitting.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.