2
$\begingroup$

I have a dataset and have the option to apply either GLM (primitive) or a Random Forest (ensemble). So far the Random Forest is giving way better results than the GLM. As it is generally believed that ensemble models should not be used unless absolutely necessary, hence I am looking for any analysis which I could perform on the dataset, which could prove that indeed the only way/better way to model the relationship between variables in the dataset is by using a ensemble model like Random Forest etc.

Improve this question
$\endgroup$
13
  • $\begingroup$ GLMs are linear in the parameters. In empirical modelling a non-linear relationship between the expected response (transformed by the link function) & a predictor is often allowed for by representing that predictor with a polynomial or spline basis: the model is still linear. Are you using "linear" in the same sense? $\endgroup$
    – Scortchi ♦
    Commented Oct 28, 2014 at 9:54
  • $\begingroup$ My understanding. Linear model: a model which has a defined number of parameters, can be either polynomial, or log or an interaction term, but the term needs to be decided beforehand. Non-Linear: no defined number of parameters. You just give in the variables, and the parameters and the number of parameters are selected when the algorithm is fitting $\endgroup$
    – show_stopper
    Commented Oct 28, 2014 at 10:05
  • $\begingroup$ @nar Linear in the parameters means the parameters exponent are all set to 1. That has nothing to do with the number of features used in the model. $\endgroup$
    – Robert Kubrick
    Commented Oct 28, 2014 at 11:39
  • $\begingroup$ Re: non-linear models should not be used unless absolutely necessary. That makes it sound as if you're saying that someone should only resort to non-linear models in extreme circumstances. $\endgroup$
    – Steve S
    Commented Oct 28, 2014 at 12:15
  • $\begingroup$ @RobertKubrick: That's not quite the distinction - e.g. $y = \frac{\beta_0 x}{\beta_1 + x} + \varepsilon$ is non-linear in the parameters $\beta_0$ & $\beta_1$. But you're right it's got nothing to do with feature selection. $\endgroup$
    – Scortchi ♦
    Commented Oct 28, 2014 at 12:56

2 Answers 2

Reset to default
1
$\begingroup$

If you want to argue that your Random Forest is generating better predictions than your linear model, then you could just show that (for example) your out-of-sample RMSE is lower for your random forest than for your linear model--it's as simple as that.

The primary downside of using a non-linear model instead of a plain-vanilla linear model is that, by and large, as your methods become more sophisticated, your resulting models will become more opaque (i.e. harder to interpret). If your goal is pure prediction then this won't matter. However, if you're trying to do some statistical inference then it's a different story.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Yes, but I think the OP's asking if there's a way of telling whether any linear model would be out-performed by random forests. (I don't think there is, given how flexible linear models can be.) $\endgroup$
    – Scortchi ♦
    Commented Oct 28, 2014 at 12:58
  • $\begingroup$ sorry for my lack of knowledge about terminologies. What I meant by linear model was : primitive models and what I meant by non-linear models was ensemble models. I have made the required edits in the question above $\endgroup$
    – show_stopper
    Commented Oct 28, 2014 at 16:09
  • $\begingroup$ @nar I'm still not sure what you're trying to say. But often the term "main effects" model is used to denote a GLM where no additional modeling (splines,interactions, regularization) were used. This is often - in my opinion incorrectly used in comparison studies - but might be what you mean. $\endgroup$
    – charles
    Commented Oct 28, 2014 at 18:28
1
$\begingroup$
  1. As it is generally believed that non-linear models should not be used unless absolutely necessary. I don't think this is accurate.
  2. GLMs are fairly flexible. Are you including regression splines, interactions... in your model?
  3. Regularization is often used with GLMs - the number of parameters don't need to be fixed.
  4. As mentioned by Scortchi, I don't think you can prove that any linear model will be outperformed by random forest - given the large number of options available.
Improve this answer
$\endgroup$
2
  • $\begingroup$ sorry for my lack of knowledge about terminologies. What I meant by linear model was : primitive models and what I meant by non-linear models was ensemble models. I have made the required edits in the question above $\endgroup$
    – show_stopper
    Commented Oct 28, 2014 at 16:08
  • 1
    $\begingroup$ @nar Not sure that distinction holds. You can have an ensemble of generalised linear models and a single non-linear model (e.g. SVM). $\endgroup$
    – conjectures
    Commented Oct 28, 2014 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.