# How to to find and evaluate optimal discretization for continuous variable with $\chi^2$ criterion?

I have a data set with continuous variable and a binary target variable (0 and 1).

I need to discretize the continuous variables (for logistic regression) with respect to the target variable and with the constrained that the frequency of observation in each interval should be balanced. I tried machine learning algorithms like Chi Merge, decision trees. Chi merge gave me intervals with very unbalanced numbers in each interval (an interval with 3 observations and another one with 1000). The decision trees were hard to interpret.

I came to the conclusion that an optimal discretization should maximise the $\chi^2$ statistic between the discretized variable and the target variable and should have intervals containing roughly the same amount of observations.

Is there an algorithm for solving this?

This how it could look like in R (def is the target variable and x the variable to be discretized). I calculated Tschuprow's $T$ to evaluate the "correlation" between the transformed and the target variable because $\chi^2$ statistics tends to increase with the number of intervals. I'm not certain if this is the right way.

Is there another way of evaluating if my discretization is optimal other than Tschuprow's $T$ (increases when number of classes decreases)?

chitest <- function(x){
interv <- cut(x, c(0, 1.6,1.9, 2.3, 2.9, max(x)), include.lowest = TRUE)
X2 <- chisq.test(df.train$def,as.numeric(interv))$statistic
#Tschuprow
Tschup <- sqrt((X2)/(nrow(df.train)*sqrt((6-1)*(2-1))))
print(list(Chi2=X2,freq=table(interv),def=sum.def,Tschuprow=Tschup))
}

• You perhaps know this; but, for the record, (1) it's neither necessary nor, often, desirable to discretize predictors for logistic regression, & (2) using the response to define the predictors introduces an optimistic bias into the estimate of your model's predictive performance, & it's therefore important to validate the whole fitting procedure (i.e. including the method you use to form the predictors). – Scortchi - Reinstate Monica Apr 23 '15 at 15:51
• Ok, how can I validate the discretization, how does this work? – Charlotte Apr 24 '15 at 7:01
• If you're using e.g. cross-validation then in each fold you don't merely carry out the logistic regression using the "optimal" discretization of the predictors determined from the whole sample, but re-calculate the "optimal" discretization as well. – Scortchi - Reinstate Monica Apr 24 '15 at 8:24