4
$\begingroup$

As an example

Let's assume we do have feature-columns $X_1$ and $X_2$ and $X_3$ and target $Y$, and it just so happens that $Y$ is the noisy spearman correlation of the three features. Would it help if you manually added the spearman correlation as a fourth feature? Or will gradient boosting 'learn' this anyway and adding it manually is not needed?

In general

  • Are there general rules of thumb which kind of feature engineering helps gradient boosting and which doesn't?
  • When should one stop adding features? Is there a maximum number of features (dimensionality) where gradient boosting starts to lose effectiveness?
Improve this question
$\endgroup$
3
  • $\begingroup$ Gradient boosting of what kind of model? Trees, CNN, RNN? $\endgroup$
    – Jon Nordby
    Commented Jun 28, 2018 at 8:35
  • $\begingroup$ Gradient boosting of trees. $\endgroup$
    – Denwid
    Commented Jun 28, 2018 at 9:46
  • $\begingroup$ Assuming ones has only three explanatory variables, a GBM is probably going to be suboptimal unless there are tens of thousands of training points; a standard GLM/GAM would probably be adequate. That said, given that any new features added contain non-redundant information, there is absolutely no reason why they would be harmful. GBM are able to do "automatic variable selection" internally but omitting information in the hope it will be "picked up" is not constructive. $\endgroup$
    – usεr11852
    Commented Jun 28, 2018 at 19:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.