I have never used Random Forests, but I have read some about it. Until now I have used GLM/GAMLSS extensively.

I would like to know:

  1. What are the advantages that RF provides over GLM/GAMLSS?
  2. What are the disadvantages of using Random Forests?

I am starting this new study which has around 25 predictor variables, and I was wondering if I should check random forests.

One disadvantage that I could find: Random Forests, at least the popular applications, are non-updatable. Is this True? This is important for me as I will run this on some real-time data and would need to assimilate new information.

  • 2
    $\begingroup$ RFs are good for automatic discovery of interactions. You could compare it to glm + stepwise addition of interaction terms. Conversely standard rf is bad for linear relationship: approximating a straight line by step functions $\endgroup$
    – seanv507
    Nov 4 '16 at 7:09

You should try lots of models. The 'no free lunch' theorem states that there is no one best model - every situation is different. Logistic regression for example is desirable when it works because parameters are very interpretable. Random forests are great because they can deal with very difficult patterns, but forget about interpreting them.

The point is - never stick to just one approach.

  • $\begingroup$ By 'interpretable', do you mean it's relationship with prediction variable is well-understood? $\endgroup$ Nov 3 '16 at 22:39
  • $\begingroup$ In a LR, each parameter value besides intercept reflects the change in log-odds of being in group 1 or group 2, with intercept being base rate. You can this understand the direct influence, and direction, of each variable. With RF, it just works, but it's hard to say why. $\endgroup$
    – HEITZ
    Nov 3 '16 at 22:46
  • 7
    $\begingroup$ Trying lots of models & picking the one that performs best - by some measure or other of predictive performance - is itself an approach to which the no free lunch theorem applies. $\endgroup$ Nov 3 '16 at 22:51
  • 1
    $\begingroup$ @HEITZ I would like to have a more detailed answer than RF just works. When it works better and all. I think understanding the guts of the model is very important and that I can do only when I know which models are superior/more applicable in which cases. Sure I could try all algos, but would like to know about the discoveries of people who have applied it to diverse situations. $\endgroup$ Nov 4 '16 at 0:47

One point to consider is are you interested in making predictions or understanding associations and carrying out inference (confidence intervals around effects). Although random forests provide a variable-importance summary, this technique is primarily aimed at prediction; there is no inference. Many researchers think they are interested in making predictions, but often there is a mismatch with their goals.

With that said, you can make predictions with glm and gamlss. You also have the flexibility of regression. A benefit to the random forests is that you do not have to specify aspects such as interactions and because of this, it may discover patterns in your data. Further it handles the case where there are more predictors than observations.

Still, there is evidence that techniques such as random forests do not use the data as efficiently as conventional techniques do. More research is needed.

I think both techniques are updatable in the same manner.

Reference: van der Ploeg, T., Austin, P. C., & Steyerberg, E. W. (2014). Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC medical research methodology, 14(1), 137.

  • $\begingroup$ Could you add an explicit reference in case the link goes dead in the future? Thanks! $\endgroup$ Nov 4 '16 at 20:02

Let's have some simple examples to show the differences. Our example have a single independent variable x and a single dependent variable - either real y or categorical z:

x    y    z
0    0.01 A
1    1.98 A
2.01 4.02 B
2.99 6.01 B

One can see that y grows as x grows and that z=B for values of x grater than something around 1.5. That is an example when GLM and similar methods rock. It is easy to make good predictions for both y and z for a new value of x. For example, for x=0.5 you would predict y=1 and z=A ("You can understand the direct influence, and direction, of each variable" as @HEITZ wrote)

x    y    z
0    0.01 A
1    1.98 B
2.01 0    A
2.99 2.01 B
4    0    A
5.01 2.00 B

We can see a clear pattern in the data again, however GLM and similar methods cannot, the connection between x and y or z is not linear nor even additive. That is when other methods as random forests needs to be used. Prediction based on GLM for x=3 would be y=1 and more or less randomly z=A or z=B. However RF or similar methods can predict what we would expect: y=2 and z=B.

Generally I would do:

  1. Try GLM and similar, if they model the data well, use it, because you can understand the model well; if not
  2. Try other methods as RF (or neural networks, etc.)
  • $\begingroup$ Thanks. GAMLSS has a large number of distributions that capable of modeling non-linear relationships too. Won't first one be more applicable for linear regression? Thanks for your insight. GLM>RF seems like a good path to follow. $\endgroup$ Nov 4 '16 at 0:49
  • $\begingroup$ @maximusdooku non-linearity is not the biggest problem, more problematic are non-monotonousness and non-additiveness of the variables $\endgroup$ Nov 4 '16 at 0:55
  • $\begingroup$ Thanks. GLMs/GAMLSS can model both additive and multiplicative relationships. Though I am not sure if they are better or worse than Random Forests in general and in what situations. $\endgroup$ Nov 4 '16 at 1:08
  • $\begingroup$ Imagine this example: values of a categorical dependent variable z=A if values of independent variables x and y are in [-1,1], z=B otherwise (As are in the square close to (0,0) and Bs are around). This is a simple example, which is hard to model by GLM and similar methods. For such complex relations are better other methods as RT. $\endgroup$ Nov 4 '16 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.