I have a dataset and have the option to apply either GLM (primitive) or a Random Forest (ensemble). So far the Random Forest is giving way better results than the GLM. As it is generally believed that ensemble models should not be used unless absolutely necessary, hence I am looking for any analysis which I could perform on the dataset, which could prove that indeed the only way/better way to model the relationship between variables in the dataset is by using a ensemble model like Random Forest etc.

  • GLMs are linear in the parameters. In empirical modelling a non-linear relationship between the expected response (transformed by the link function) & a predictor is often allowed for by representing that predictor with a polynomial or spline basis: the model is still linear. Are you using "linear" in the same sense? – Scortchi Oct 28 '14 at 9:54
  • My understanding. Linear model: a model which has a defined number of parameters, can be either polynomial, or log or an interaction term, but the term needs to be decided beforehand. Non-Linear: no defined number of parameters. You just give in the variables, and the parameters and the number of parameters are selected when the algorithm is fitting – show_stopper Oct 28 '14 at 10:05
  • @nar Linear in the parameters means the parameters exponent are all set to 1. That has nothing to do with the number of features used in the model. – Robert Kubrick Oct 28 '14 at 11:39
  • Re: non-linear models should not be used unless absolutely necessary. That makes it sound as if you're saying that someone should only resort to non-linear models in extreme circumstances. – Steve S Oct 28 '14 at 12:15
  • @RobertKubrick: That's not quite the distinction - e.g. $y = \frac{\beta_0 x}{\beta_1 + x} + \varepsilon$ is non-linear in the parameters $\beta_0$ & $\beta_1$. But you're right it's got nothing to do with feature selection. – Scortchi Oct 28 '14 at 12:56

If you want to argue that your Random Forest is generating better predictions than your linear model, then you could just show that (for example) your out-of-sample RMSE is lower for your random forest than for your linear model--it's as simple as that.

The primary downside of using a non-linear model instead of a plain-vanilla linear model is that, by and large, as your methods become more sophisticated, your resulting models will become more opaque (i.e. harder to interpret). If your goal is pure prediction then this won't matter. However, if you're trying to do some statistical inference then it's a different story.

  • 1
    Yes, but I think the OP's asking if there's a way of telling whether any linear model would be out-performed by random forests. (I don't think there is, given how flexible linear models can be.) – Scortchi Oct 28 '14 at 12:58
  • sorry for my lack of knowledge about terminologies. What I meant by linear model was : primitive models and what I meant by non-linear models was ensemble models. I have made the required edits in the question above – show_stopper Oct 28 '14 at 16:09
  • @nar I'm still not sure what you're trying to say. But often the term "main effects" model is used to denote a GLM where no additional modeling (splines,interactions, regularization) were used. This is often - in my opinion incorrectly used in comparison studies - but might be what you mean. – charles Oct 28 '14 at 18:28
  1. As it is generally believed that non-linear models should not be used unless absolutely necessary. I don't think this is accurate.
  2. GLMs are fairly flexible. Are you including regression splines, interactions... in your model?
  3. Regularization is often used with GLMs - the number of parameters don't need to be fixed.
  4. As mentioned by Scortchi, I don't think you can prove that any linear model will be outperformed by random forest - given the large number of options available.
  • sorry for my lack of knowledge about terminologies. What I meant by linear model was : primitive models and what I meant by non-linear models was ensemble models. I have made the required edits in the question above – show_stopper Oct 28 '14 at 16:08
  • 1
    @nar Not sure that distinction holds. You can have an ensemble of generalised linear models and a single non-linear model (e.g. SVM). – conjectures Oct 28 '14 at 16:13

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.