0
$\begingroup$

I might have a silly question - I am building a linear model with many many attributes. I have narrow down to a few - I do have a group of 3 attributes that are highly correlated (for example sales amount for the past 1 years, 2 years and 3 years). I don't want to only keep one of them and exclude the rest.

Can I build a decision tree of just those 3 attributes against the target, and based on the tree results ( the rules) and create a new attribute combing those 3? so it will be a binned attributes based on the tree nodes.

it is very predictive and utilized all 3 original attributes. I am wondering whether there is anything major wrong with doing this? I cannot find anyone doing things like this.

Thank you!

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ @MichaelM Hi, the end goal is to create a linear model which is explainable. i was thinking create one of the variable of the linear model using tree. $\endgroup$ – zhifff Sep 1 '20 at 17:52
  • 1
    $\begingroup$ Feature engineering based on the response is problematic as you can't trust insample results of the linear model (p values, R-squared etc) anymore. $\endgroup$ – Michael M Sep 1 '20 at 18:01
  • $\begingroup$ @MichaelM isnt WOE transformation based on the response? $\endgroup$ – zhifff Sep 1 '20 at 18:19
  • $\begingroup$ The same problem applies to WOE. An option in your situation is to use PCA or similar to reduce the three correlated variables to just one and work with this. $\endgroup$ – Michael M Sep 1 '20 at 18:22
0
$\begingroup$

Go look for Basis expansion and additive models, additive model is basically a linear combination of basis functions, shares the same idea as yours. In your case the basis functions are decision trees, boosting model with trees can solve your problem well.

The book Elements of Statistical Learning has a very good explanation of basis expansion, additive models and boosting methods.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.