8
$\begingroup$

For example, when I use the xgboost algorithm, there are two continuous variables X1 and X2, do I need to specify the product X1*X2 explicitly at the beginning? Or the algorithm can automatically pick up the effect of the X1*X2?

$\endgroup$
2
  • 1
    $\begingroup$ Welcome to the site! Trees can consider high-order interactions automatically. $\endgroup$
    – Randel
    Commented Jan 23, 2016 at 21:11
  • 1
    $\begingroup$ Just make sure your max_depth is at least 2 to find 2-way interactions. $\endgroup$
    – Dex Groves
    Commented Sep 2, 2016 at 22:35

2 Answers 2

4
$\begingroup$

Yes and no. By having a sufficiently deep tree (at least two splits deep) and splitting on both $x_1$ and $x_2$, tree based model like xgboost (or LightGBM or catboost) can eventually approximate (given enough data) any relationship between $x_1\times x_2$ and your prediction target of interest. Of course, if you know that some function $f(x_1\times x_2)$ is a predictor of your outcome of interest, then specifying a feature that is that exact transformation of $x_1$ and $x_2$ is what you should do.

What to do in practice? If you have a lot of data and are not that sure that the interaction matters, perhaps don't specify it as a feature. If you don't have a lot of data and suspect it's an important feature, you probably want to put this feature into xgboost. If you are inbetween - try it out using an appropriate cross-validation scheme.

$\endgroup$
0
$\begingroup$

The answer to your question depends on what class of split rules you allow in the fitting of a decision tree. If the only class of allowable splits are on a single variable you will never be able to capture the interaction behavior described in the post. In fact what you will see that allows you to diagnose something like this has occurred is a repetition of splits in the decision tree on the two interacting variables and the repeated splits may occur several times. If you allow classes of splitting rules that allow for polynomials up to the order of the interaction you think may occur in your data (here 2nd order) then you will be able to capture the behavior in the decision tree that is fit to the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.