0
$\begingroup$

I am trying to discretize a continuous variable that does not follow a linear relationship with our response variable. I am trying to find the optimal way to discretize this variable to better capture the nonlinear relationship. The method we are using right now is to cut the variable into numerous quantiles, then conducting t-tests between each neighboring group, and combining the most like neighbors until the highest p-value is below our significance level. Is there any more efficient or accurate way to conduct this process? Many of the discretization methods I have been able to find seem to refer to a binary response.

$\endgroup$
2
$\begingroup$

Don't discretize at all, see Frank Harrell's explanation why this is a bad idea. If you have a nonlinear relationship, use instead, e.g., restricted cubic splines. Frank Harrell's textbook Regression Modeling Strategies contains a very good introduction to these.

$\endgroup$
  • $\begingroup$ I understand now the reasoning for why discretizing is not a great idea, but the models constructed with bins are out-performing all models with cubic splines in glm using ns() and smoothing in mgcv. Any idea why this would be occurring? And the aspect of non-linear regression that initially kept us from embracing splines/smoothing is the loss of interpretability. $\endgroup$ – J. Gursky Aug 8 '17 at 21:04
  • $\begingroup$ How exactly do they outperform? In prediction on holdout data? If so, are the holdout data binned or not, and what is the accuracy measure? $\endgroup$ – Stephan Kolassa Aug 9 '17 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.