4
$\begingroup$

In GAMs interaction terms have to be expressly specified as covariates, even for simple linear relationships. On the contrary, with Gradient boosting this is not nesessary because the algorithm itself does it implicitly by design. This last point is not clear to me.

Hence: how does gradient boosting include interaction terms?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Gradient boosting fits a GAM. A GAM does not have to be parametric. $\endgroup$
    – Simon Boge Brant
    Sep 18, 2019 at 12:03
  • $\begingroup$ How can you fit a GAM without assuming a distribution? $\endgroup$
    – OnlyAL
    Sep 18, 2019 at 12:18
  • 1
    $\begingroup$ In general you do not necessarily assume a distribution when you fit any regression model, just something about the regression function (usually the conditional mean of $Y$ given the covariates $X$, possibly transformed via a link function). If you assume that the additive components that make up the regression function are nonparametric (e.g. regression trees or splines) then the GAM is a nonparametric model. $\endgroup$
    – Simon Boge Brant
    Sep 18, 2019 at 12:51
  • $\begingroup$ This would depend on the method. E.g., You could find out how trees do this. But you'd have to ask about each method separately $\endgroup$
    – Peter Flom
    Sep 18, 2019 at 13:31

1 Answer 1

Reset to default
3
$\begingroup$

Of course, the specifics will depend on your particular nonparametric method. Gradient boosting is a meta-method, and it typically uses classification and regression trees (CARTs), so I'll work with that.

An interaction is of the following abstract form:

The impact of a predictor A on the outcome depends on the value of a different predictor B.

CARTs can easily model this by first splitting on B and then, on lower levels, splitting on A differently for low values of B than for high values of B.

If you have enough data, this will indeed happen automatically. Since CARTs consider all possible interactions (instead of the user having to explicitly model them), they need much more data to avoid fitting noise, i.e., spurious interactions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ True, but for every split, the relationship between the predictors will only be linear. How do we then model interactions with other functional forms (say multiplicative)? Do we just represent it as a new predictor? $\endgroup$
    – OnlyAL
    Sep 18, 2019 at 12:14
  • 1
    $\begingroup$ Multiplicative interactions will be approximated by locally constant fits. (Or linear ones. It depends on your CART.) Your CART will split each predictor multiple times to get multiple locally constant fits. If you have enough data, you can approximate multiplicative (or any other) relationships arbitrarily closely. $\endgroup$
    – Stephan Kolassa
    Sep 18, 2019 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.