0
$\begingroup$

I have a predictor that exhibits "bimodal" behaviour. How can I engineer this feature to improve performance within a Decision Tree?


For an intuitive example, consider how a binary flag of "moves house in next 6 months?" may correlate with a continuous variable of "change in income over last month". This may yield a distribution like the following:

binned data to illustrate bimodal nature of continuous data (continuous data has been binned; simulated to illustrate bimodal nature of data)

  • those that get larger increase in income (e.g., get promoted) are more likely to move house (e.g., up-size) than those whose income remains unchanged
  • those that get larger decreases in income (e.g., made redundant) are also more likely to move house (e.g., move back in with parents)

My trouble is that when using a Decision Tree (non-negotiable), cuts are applied on nodes to find the best value at which to split. However, due to the bimodal nature of the feature, the cuts cannot successfully split the data well enough.

Question: how can I engineer this feature to improve performance within a Decision Tree?


I have considered:

  • adding an additional feature that states whether the change is positive or negative
  • converting the data into categories (bins), which can later be one-hot-encoded
$\endgroup$
  • $\begingroup$ Decision trees are known to have sub-par accuracy precisely beucase of their simple rules. Have you tried some other method to compare results? $\endgroup$ – user2974951 Feb 5 at 7:58
  • $\begingroup$ Due to technical limitations, our company needs the model to be translatable into "simple" code - I think this limits us to Decision Trees (i.e., conditional IF-ELSE logic) or Logistic Regression (i.e., a simple linear combination of [scaled] predictors). However, from what I understand about these algorithms under-the-hood, they can not cope very well with features that have "complex" structure (i.e., in my example, the distribution is U-shaped, and not monotonically increasing). Can you think of any techniques to re-engineer this feature? $\endgroup$ – Ben Feb 5 at 10:55
  • $\begingroup$ I don't really have an answer for you, except maybe trying to recode your variables to avoid this situation. For ex. add an additional variable (bought new house, moved to parents) if you have such information. Or any other such information that can be used to separate into better groups. $\endgroup$ – user2974951 Feb 7 at 8:15
  • $\begingroup$ Yep - I got that impression from the lack of answers! Of course, more features would solve the problem, but sometimes it's not possible (due to lack of resources) and we have to make do. $\endgroup$ – Ben Feb 8 at 7:06
  • $\begingroup$ I came up with a solution - it's not perfect, but it has made the data more usable than it was $\endgroup$ – Ben Feb 8 at 7:21
0
$\begingroup$

Take the absolute value of the feature. So instead of a feature of

$$x_1$$

you'd have

$$\mid x_1 \mid$$

With pandas, this can be done using df[feature].abs()


Note that this assumes the data is symmetric about $0$, i.e., the probability density function for $x>0$ is $f(x)$, and for $x<0$ is $f(-x)$; if this assumption is true, then this will make the data far more linearly separable than in its current form.

$\endgroup$
  • $\begingroup$ This might work, but this completely changes the interpretation of the variable. You lost important information, since you cannot distinguish who's change in income is positive or negative anymore. $\endgroup$ – user2974951 Feb 8 at 7:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.