Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm wondering if there exist methods similar to one used in random forest algorithm - I mean taking simultaneously bootstrap sample and random subset of features, then building statistisal model. Have anyone took this approach for building set of regression models ? Is this approach ( random subsample plus random subset of features ) somehow universal ?

Edition : the question is all about the possibility of putting some other model in the place of classification tree in random forest, what is left is some kind of meta-algorithm (as bagging can be view as meta-algorithm) = bagging + random subsets of variables

share|improve this question
You can use random forest for regression - if by regression you mean a continuous target. – B_Miner Jan 8 '13 at 23:44
What you are probably looking for is known as random subspaces (RS). This is the universal approach that has been published even before the random forest (RF). However, RF proposed using out of the bag samples for building the ensemble. – soufanom Jan 9 '13 at 3:57
@B_Miner yes I know it, the question is all about the possibility of putting some other model in the place of classification tree in random forest – Qbik Jan 10 '13 at 13:07

I believe what you are looking for is known as "bagging", a portmanteau of "boostrap aggregation". Indeed, the Random Forest algorithm is sometimes referred to as the bagged trees algorithm. There are various implementations of this, for instance the R library CARET provides a framework for creating bagging models from any base model of your choice (such as SVM, logistic regression or whatever else you like).

share|improve this answer
but bagging is only about sampling from training set but not from the set of variables – Qbik Jan 10 '13 at 13:05
That's not the use of the term that I've encountered, which includes R packages for bagging and the journal articles written by the bagging method pioneer Leo Brieman. I'm sure people have done bagging without taking sub-samples of the variables, however the term refers in general to methods that take random samples of the training set and predictors, at least in my experience. Again I'd point out that the RF algorithm is also known as 'Bagged Trees' and this certainly includes, in general, sampling the variables. – Bogdanovist Jan 10 '13 at 19:52

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.