157

$\text{error = bias + variance}$ Boosting is based on weak learners (high bias, low variance). In terms of decision trees, weak learners are shallow trees, sometimes even as small as decision stumps (trees with two leaves). Boosting reduces error mainly by reducing bias (and also to some extent variance, by aggregating the output from many models). On the ...


122

Random Forests are hardly a black box. They are based on decision trees, which are very easy to interpret: #Setup a binary classification problem require(randomForest) data(iris) set.seed(1) dat <- iris dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other')) trainrows <- runif(nrow(dat)) > 0.3 train <- dat[trainrows,] ...


85

Random Forest is a bagging algorithm rather than a boosting algorithm. They are two opposite way to achieve a low error. We know that error can be composited from bias and variance. A too complex model has low bias but large variance, while a too simple model has low variance but large bias, both leading a high error but two different reasons. As a result, ...


72

Random forest uses bagging (picking a sample of observations rather than all of them) and random subspace method (picking a sample of features rather than all of them, in other words - attribute bagging) to grow a tree. If the number of observations is large, but the number of trees is too small, then some observations will be predicted only once or even not ...


72

Since RF can handle non-linearity but can't provide coefficients, would it be wise to use Random Forest to gather the most important Features and then plug those features into a Multiple Linear Regression model in order to explain their signs? I interpret OP's one-sentence question to mean that OP wishes to understand the desirability of the following ...


59

This question is addressed in this very nice post. Please take a look at it and the references therein. http://fastml.com/what-is-better-gradient-boosted-trees-or-random-forest/ Notice in the article that the speaks about calibration, and links to another (nice) blog post about it. Still, I find that the paper Obtaining Calibrated Probabilities from ...


53

Some time ago I had to justify a RF model-fit to some chemists in my company. I spent quite time trying different visualization techniques. During the process, I accidentally also came up with some new techniques which I put into an R package (forestFloor) specifically for random forest visualizations. The classical approach are partial dependence plots ...


47

First (and easiest) solution: If you are not keen to stick with classical RF, as implemented in Andy Liaw's randomForest, you can try the party package which provides a different implementation of the original RF™ algorithm (use of conditional trees and aggregation scheme based on units weight average). Then, as reported on this R-help post, you can ...


46

First I would like to clarify what the importance metric actually measures. MeanDecreaseGini is a measure of variable importance based on the Gini impurity index used for the calculation of splits during training. A common misconception is that the variable importance metric refers to the Gini used for asserting model performance which is closely related to ...


45

This is a common rookie error when using RF models (I'll put my hand up as a previous perpetrator). The forest that you build using the training set will in many cases fit the training data almost perfectly (as you are finding) when considered in totality. However, as the algorithm builds the forest it remembers the out-of-bag (OOB) prediction error, which ...


43

I'm not an authoritative figure, so consider these brief practitioner notes: More trees is always better with diminishing returns. Deeper trees are almost always better subject to requiring more trees for similar performance. The above two points are directly a result of the bias-variance tradeoff. Deeper trees reduces the bias; more trees reduces the ...


41

This is partly a response to @Sashikanth Dareddy (since it will not fit in a comment) and partly a response to the original post. Remember what a prediction interval is, it is an interval or set of values where we predict that future observations will lie. Generally the prediction interval has 2 main pieces that determine its width, a piece representing ...


40

I'm four years late, but if you really want to stick to the randomForest package (and there are some good reasons to do so), and want to actually visualize the tree, you can use the reprtree package. The package is not super well documented (you can find the docs here), but everything is pretty straightforward. To install the package refer to initialize.R ...


37

There must be some features in your training set with class 'char' . Please check this > a <- c("1", "2",letters[1:5], "3") > as.numeric(a) [1] 1 2 NA NA NA NA NA 3 Warning message: NAs introduced by coercion


37

It's common to find code snippets that treat $T$ as a hyper-parameter, and attempt to optimize over it in the same way as any other hyper-parameter. This is just wasting computational power: when all other hyper-parameters are fixed, the model’s loss stochastically decreases as the number of trees increases. Intuitive explanation Each tree in a random ...


36

Gradient Boosting Trees uses CART trees (in a standard setup, as it was proposed by its authors). CART trees are also used in Random Forests. What @user777 said is true, that RF trees handles missing values either by imputation with average, either by rough average/mode, either by an averaging/mode based on proximities. These methods were proposed by Breiman ...


34

How are you getting that 99% AUC on your training data? Be aware that there's a difference between predict(model) and predict(model, newdata=train) when getting predictions for the training dataset. The first option gets the out-of-bag predictions from the random forest. This is generally what you want, when comparing predicted values to actuals on the ...


33

Thanks for a very good question! I will try to give my intuition behind it. In order to understand this, remember the "ingredients" of random forest classifier (there are some modifications, but this is the general pipeline): At each step of building individual tree we find the best split of data While building a tree we use not the whole dataset, but ...


31

To avoid over-fitting in random forest, the main thing you need to do is optimize a tuning parameter that governs the number of features that are randomly chosen to grow each tree from the bootstrapped data. Typically, you do this via $k$-fold cross-validation, where $k \in \{5, 10\}$, and choose the tuning parameter that minimizes test sample prediction ...


31

The Extra-(Randomized)-Trees (ET) article contains a bias-variance analysis. On page 16 you can see a comparison with multiple methods including RF on six tests (tree classification and three regression). Both methods are about the same, with the ET being a bit worse when there is a high number of noisy features (in high dimensional data-sets). That said, ...


30

According to cosine theorem, in euclidean space the (euclidean) squared distance between two points (vectors) 1 and 2 is $d_{12}^2 = h_1^2+h_2^2-2h_1h_2\cos\phi$. Squared lengths $h_1^2$ and $h_2^2$ are the sums of squared coordinates of points 1 and 2, respectively (they are the pythagorean hypotenuses). Quantity $h_1h_2\cos\phi$ is called scalar product (= ...


30

%IncMSE is the most robust and informative measure. It is the increase in mse of predictions(estimated with out-of-bag-CV) as a result of variable j being permuted(values randomly shuffled). grow regression forest. Compute OOB-mse, name this mse0. for 1 to j var: permute values of column j, then predict and compute OOB-mse(j) %IncMSE of j'th is (mse(j)-mse0)...


30

No. Random Forests are based on tree partitioning algorithms. As such, there's no analogue to a coefficient one obtain in general regression strategies, which would depend on the units of the independent variables. Instead, one obtain a collection of partition rules, basically a decision given a threshold, and this shouldn't change with scaling. In other ...


29

The answer to your question depends on what similarity/distance function you plan to use (in SVMs). If it's simple (unweighted) Euclidean distance, then if you don't normalize your data you are unwittingly giving some features more importance than others. For example, if your first dimension ranges from 0-10, and second dimension from 0-1, a difference of ...


29

This thread refers to two other threads and a fine article on this matter. It seems classweighting and downsampling are equally good. I use downsampling as described below. Remember the training set must be large as only 1% will characterize the rare class. Less than 25~50 samples of this class probably will be problematic. Few samples characterizing the ...


27

I don't know exactly what you did, so your source code would help me to guess less. Many random forests are essentially windows within which the average is assumed to represent the system. It is an over-glorified CAR-tree. Lets say you have a two-leaf CAR-tree. Your data will be split into two piles. The (constant) output of each pile will be its ...


27

A random forest is not considered a boosting type of algorithm. As explained in your boosting link: ...most boosting algorithms consist of iteratively learning weak classifiers with respect to a distribution and adding them to a final strong classifier. When they are added, they are typically weighted in some way that is usually related to the weak ...


26

"Is this a statement about the feature as a whole or about specific values within the feature?" "Global" variable importance is the mean decrease of accuracy over all out-of-bag cross validated predictions, when a given variable is permuted after training, but before prediction. "Global" is implicit. Local variable importance is the mean decrease of ...


Only top voted, non community-wiki answers of a minimum length are eligible