I've been using RFs for about a month now (no experience before). What I've noticed so far that if I keep the configuration of a RF (number of trees, objective function, split conditions, etc) and rerun on the same data, I get very different results every time I do it: false positives and true positives vary from trial to trial quite a bit, which is annoying because I want a stable, predictable result. Is there any way to control it? Intuitively increase in tree number should take care of the problem, but it doesn't. I'd appreciate suggestions on this.

  • 3
    $\begingroup$ The random in the name is meaningful here. To get the same result every time on the same data, you need to make identical random choices. In R you can achieve this with set.seed() and other languages may have something similar $\endgroup$ – Henry Sep 30 '16 at 14:28
  • $\begingroup$ well I never said 'identical', didn't I? I mean more of a variance reduction, i.e. predictable result. I increased the number of trees, but it didn't help. $\endgroup$ – Alex Sep 30 '16 at 14:31
  • $\begingroup$ @Sycorax: thanks, I'll try. 1) Could you point me to a publication that examines it (unless it's purely your experience though) 2) I don't immediately see why increasing the min leaf size leads to variance reduction. If you answer the question, I'll upvote and accept it $\endgroup$ – Alex Sep 30 '16 at 14:48
  • $\begingroup$ How many features and samples you have? $\endgroup$ – Firebug Sep 30 '16 at 14:59
  • 2
    $\begingroup$ Feature selection is mostly useless with Random Forests. Try without it. Also, again, how many samples and features you have, in total? $\endgroup$ – Firebug Sep 30 '16 at 15:28

Two thoughts.

  1. RF are often split until purity. This often means that there are many terminal nodes, each with a single observation. The final splits leading up to these nodes may not generalize very well because there are so few observations to work with at that depth of the tree. So you may get more generalizable trees with lower out-of-sample variance if you increase the minimum node size from 1 to something like 10 or more (depending on how much data you have -- this is another hyper-parameter, so you might profitably tune it). This also has the property of yielding consistent probability estimates, which can be desirable in some contexts.

  2. Increasing the number of trees will reduce the variance of the estimator. This is an obvious consequence of one of the CLTs -- each tree is a binomial trial, and the prediction of the forest is the average of many binomial trials. Moreover, the trees are iid in the sense that they are all fit on different re-samplings of the data and different random subsets of features. So you have iid binomial trails (which have finite variance because each trial is 0 or 1, i.e. has finite cardinality). This can make the predictions less volatile because the trees only have to explain chunks of your data, instead of each observation. So four times as many trials will cut the standard error of the mean in half.

There is extended discussion of some of these RF properties in Elements of Statistical Learning. The consistency property is discussed in Malley JD, Kruppa J, Dasgupta A, Malley KG, Ziegler A. Probability Machines: Consistent Probability Estimation Using Nonparametric Learning Machines. Methods of Information in Medicine. 2012;51(1):74-81. doi:10.3414/ME00-01-0052.

Finally, as a general observation, the best regularizer is more data, and better features usually beat a cleverer algorithm.

  • $\begingroup$ Perhaps I'm confused about the way RFs work: there's just not way to get a consistent result out of them, just because they are supposed to give different results every run? $\endgroup$ – Alex Oct 1 '16 at 18:14
  • $\begingroup$ @Alex It depends on what you mean by consistent. If you fix the seed, the results are completely determined. But "consistency" in a statistical sense has a very specific meaning as defined in the Malley et al. paper. $\endgroup$ – Sycorax says Reinstate Monica Oct 1 '16 at 18:41
  • $\begingroup$ No fixing the seed is not what I want. Finding parameters that yield an algorithm that produces low-variance result is. $\endgroup$ – Alex Oct 1 '16 at 18:48
  • $\begingroup$ What I didn't mention is that I'm working with an object tracking data, a time series of sorts. Maybe this changes something. $\endgroup$ – Alex Oct 1 '16 at 18:49
  • $\begingroup$ @Alex I've addressed how to reduce the variance of random forest in my post. $\endgroup$ – Sycorax says Reinstate Monica Oct 1 '16 at 19:05

I mean more of a variance reduction, i.e. predictable result. I increased the number of trees, but it didn't help.

Due to the central limit theorem, and due to the fact Random Forests predictions are obtained through averaging, increasing the number of trees should help. The default in R is 500L, set this as high as you can support (I've often put it to 5000L, depending on the data).

The randomness in Random Forests come from both attribute bagging and bootstrap aggregating. You may also try to reduce the randomness either of those add.

Last, depending on how many features and how many samples you have, it might simply due to the data, and no amount of hyperparameter tinkering will solve it.

You mentioned in a comment:

I've tried quite a few, between 6 and 400. I use a regression-based feature selector that keeps all features with p-value below 0.05

And, as I said, I find feature selection mostly useless with Random Forests. The reasons are simple: you risk overfitting by doing that selection and Random Forests are good with large number of features. Let the forest decide which features are worthy, unless you have orders of magnitude more features than samples, then do some small reduction, just enough to remove noise features.

  • $\begingroup$ yeah, I'm trying this too. Specifically I vary the share of positives in the data, anywhere between 0.3 and 0.1. I notice that higher proportions of positives usually leads to both more true and false posivites. But still, even with this observation the result is barely more predictable. $\endgroup$ – Alex Sep 30 '16 at 15:13
  • $\begingroup$ why is feature selection useless with RFs? $\endgroup$ – Alex Oct 1 '16 at 17:07

Adding to what users said, if you average many uncorrelated or little correlated variable you get a reduction in variance.

Define $X=\sum_{i}^{B}T_i$ where $T_i$ is a single decision tree, B number of decision tree.
$Var(X)=\frac{1}{B^2}\sum_i\sum_jcov(X_i,X_j)=\frac{1}{B^2} \sum_{i} ( \sum_{i\neq j} Cov(X_i,X_j)+Var(X_i)) = \frac{1}{B^2}\sum_ i ( (B-1)\sigma^2\rho +\sigma^3)=\frac{B (B-1)\sigma^2\rho+B\sigma^2}{B^2}=\rho\sigma^2+\sigma^2\frac{1-\rho}{B}$.

First quantity decreases if $\rho$ decreases to zero ( uncorrelation) and second quantity decreases if B increases.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.