4
$\begingroup$

I'm doing some research into methods for discretizing a continuous variable coupled with a binary target variable to find the optimal split points to maxamise a measure of impurity (gini/entropy).

First off I'm having some trouble coming up with google-able terms. "Optimal Split" seems to relate to choosing which variable to split on in a decision tree as opposed to how to split that variable up. Is there a defined title on this problem?

I imagine to convert a continuous variable into a binary variable I could potentially try setting a split point at each distinct value in the data and take the one that returns the maximum entropy or gini.

But to extend it beyond binary splits the search space would grow to make that method pretty expensive.

Are there any popular methods for solving this?

Improve this question
$\endgroup$
2
  • $\begingroup$ Tree methods will usually do exactly this: find a sequence of splits on the predictor variable(s) so that the impurity of the classification is minimized. Gini and entropy are commonly used. Of course trees will work in a greedy way and therefore not be guaranteed to find the global optimum. That said, is there any other reason why a standard tree would not be the solution? $\endgroup$
    – Stephan Kolassa
    Mar 13, 2014 at 10:28
  • $\begingroup$ Standard trees do not handle continuous predictors well. $\endgroup$
    – Frank Harrell
    Mar 13, 2014 at 10:39

1 Answer 1

Reset to default
5
$\begingroup$

Your question raises so many issues that it is difficult to know where to start. First of all you need to make sure that the accuracy score you wish to optimize is a proper scoring rule, i.e., that it is not optimized by a bogus model using the wrong features. Second, it is rare in nature to have discontinuities in predictors other than time. Third, when there are no true discontinuities in predictors, any algorithm that attempts to find such cutpoints will yield answers that other analytical methods or other datasets will surely disagree with.

The literature on the horrendous problems of dichotomizing continuous variables is now rather mature. One overview may be found at http://biostat.mc.vanderbilt.edu/CatContinuous . There are many methods for keeping continuous variable continuous, e.g., regression splines, random forests.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the reply. To your points: 1) I appreciate this point, I'm sure it's possible that some scores could improve with the number of splits for eg, I'll look into this. 2) Regarding using the wrong features to optimise a model - I'm currently intending to use this for univariate analysis. So each 'model' would just be the one feature. 3) Some of my variables do have natural discontinuities as, I'm considering a hybrid approach to potentially treat those that appear monotonic differently. Can I ask how random forests keep continuous variables continuous? Does each tree not discretize? $\endgroup$
    – Ger
    Mar 13, 2014 at 11:18
  • 2
    $\begingroup$ Random forests does not do as good a job as continuous models, but by combining many trees it approximates continuous effects with step functions - better than single tree methods can do. Continuous methods are better for this. Monotonic/non-monotonic is not reallly the issue. Adding more splits in the way I think you describe will increase variance which decreases the probability of predicting close to the "truth". $\endgroup$
    – Frank Harrell
    Mar 13, 2014 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.