4
$\begingroup$

Currently my team is using a logistic regression model with pre-computed feature crosses, to predict a binary outcome. So, given data with columns A and B with values {A1, A2} and {B1, B2}, there is a step of creating new feature AxB: {(A1, B1), (A1, B2), (A2, B1), (A2, B2)} and using that in the LR model.

One colleague wants to use XGBoost instead, and has made the statement that with XGBoost such pre-computation of crosses is not necessary. However, this is not my understanding of how tree-based methods work; I thought that the trees can only place cuts orthogonally on columns in the dataset, and are therefore bad at finding relationships such as AB, A/B, B/A, which may be computed with a prior feature engineering step.

A second colleague who is more familiar with the algorithm has said that pre-computing these features can still be helpful when using XGBoost, since otherwise the trees must be grown very deeply and the run time becomes prohibitive. Still, I am confused by this because again, it's not in my understanding at all that a tree-based algorithm would find crossed features.

Can anyone explain whether XGBoost can really do this, and if so, is the ability dependent on the hyperparameters used, such as tree depth? Are there any good references to read that cover this point?

$\endgroup$
3
$\begingroup$

Both are correct. Tree-based methods cut perpendicular to a feature axis, so if the boundary is actually a diagonal, many perpendicular splits will be used to approximate a diagonal boundary. On the one hand, trees (like xgb) can work out this boundary; on the other hand, lots of splits will be needed to accomplish it.

Which strategy (letting the computer work it out, or pre-specifying feature products) is more efficient is problem-specific. If the data are not partitioned "diagonally," or are so partitioned rarely, products of features offer no improvement, so you'll be wasting time whenever xgb considers an unhelpful product feature. On the other hand, if product feature are helpful, including them can preclude the splits that would be necessary to approximate a diagonal. Finally, if you had $p$ features originally, including all products adds $\frac{p(p-1)}{2}$ additional features, which itself may be prohibitive.

$\endgroup$
  • 1
    $\begingroup$ This was really helpful, especially the specification about how many perpendicular splits can approximate a diagonal boundary. As soon as I read that I could visualize it and wondered why I hadn't gotten it earlier. $\endgroup$ – RKD314 Nov 30 '17 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.