0
$\begingroup$

I've got two columns of data - a continuous variable that I'd like to treat as a categorical variable (i.e. bin it up), and a metric I want to measure by bin. Let's say the first column is income and second column is # of trips outside the house (ranging from 1-300, not normally distributed).

If we wanted to explain to execs how income influences # of trips, a simple approach is to bin up the various incomes in 4-5 groups (with the last group being something like $500K+), and have the greatest differences in # of trips between those bins. In other words, I want to enable 4-5 bins that most clearly separate the data (with the underlying idea that more income means more trips, roughly linearly).

The approach I've been taking is really manual - coming up with random bins, finding the standard deviation of # trips between them, and trying to find out where it's at is maximum. Does anyone have a better way to do this via code to optimize it?

$\endgroup$
2
  • 4
    $\begingroup$ Yes: it's called a spline: stats.stackexchange.com/search?q=regression+spline $\endgroup$
    – whuber
    Dec 9, 2020 at 0:23
  • $\begingroup$ What does "most clearly separate the data" mean? precisely and quantitatively. Once you can answer that, then you can try to figure out how to optimize it. $\endgroup$
    – Dave
    Feb 17, 2021 at 16:18

2 Answers 2

1
$\begingroup$

Yes, I think you are referring to the optimal binning with constraints for a continuous target. The OptBinning package solves a mixed-integer optimization problem to obtain the provably optimal binning. See: http://gnpalencia.org/optbinning/tutorials/tutorial_continuous.html.

$\endgroup$
1
  • $\begingroup$ Good to know about this package. Will try to use it for my problem. However, I have a quick question. when most of the posts here in this forum advice against binning continuous variables, can I check with you on why do you think/how this binning will be useful? As you are the creator of this package, I feel it would be useful to know your views on this. $\endgroup$
    – The Great
    Mar 7, 2022 at 4:09
0
$\begingroup$

Any binning will be suboptimal compared to (possibly nonlinear) regression. There are plenty of threads on this site and elsewhere that discuss the dangers of the dichotomization of continuous variables.

Instead of looking for some complicated, suboptimal procedure that produces results that could be explained to execs, fit the best, most principled model you can, and then look for ways to explain it to execs simply. So instead of binning your data and fitting a model to those bins, fit the model to continuous data, and then present it to execs as your model predicts # of trips is for a person with an income 10k, 20k, 50k, 80k ...

Or you can plot a graph with an income on the x axis and the number of trips on the y axis, showing pointclouds, regression line or quantile lines with some arrows, labels, and captions that explicitly explain important observations fivethirtyeight style, or you can just walk them through it. You probably don't want to just look at the average number of trips, since the number of trips will be IMO very skewed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.