How would a skewed variable impact a classification problem (logistic regression, tree model)?

Is it justified to bin the skewed variable ?

My data set comprises of younger demographic and fewer older candidates which is expected since it is about college goers.

Also, since college goers are above the age of 19 could this data be considered censored ? And what impact would this have on classification ?

Note: It is important for me to understand the impact of age if any on the classification.

While this When should we discretize/bin continuous independent variables/features and when should not? advises against binning and the references talk about the Modifiable Areal Unit problem, this Why Binning Variables in Predictive Analytics? suggests that (1) There is implicit binning in decision trees and (2) Quotes an example of the impact an optimal range of temperature has on flowering which cannot be captured by continuous variable.

In my problem I am trying to learn the impact of age among other variables on graduating and most college goers are young except a few older students.

If I am building a tree model for classification, should the age be considered a skewed variable and does binning help in building a tree or is detrimental to the process of splits using entropy or gini-index ?

Wikipedia article on Binning says:

Binning is also used in machine learning to speed up[3] the decision-tree boosting method for supervised classification and regression in algorithms such as Microsoft's LightGBM and scikit-learn's Histogram-based Gradient Boosting Classification Tree

Both posts seems to agree that there is loss of information when continuous variables are binned.

  • 3
    $\begingroup$ Binning sacrifices information and will result in a worse model generally. Tree based models do implicitly bin continuous features by making splits. However, the splits are selected for their optimality. Binning predictors arbitrarily is not the same. Binning will result in faster performance for tree based models. However, it is easy to improve speed while sacrificing information. You can remove variables or observations and achieve a significant reduction in training time. That doesn't mean it is ever recommended unless you are resource constrained. $\endgroup$ – Ryan Volpi Jul 5 '20 at 20:30

If you are using trees then the algorithm will select the bins for you, regardless of whether the variable is skewed or normal or whatever. There is no need for you to "pre-bin" and such an approach can only make the result worse.

If you are using some kind of regression (you mention logistic regression) then you can use a spline of a continuous variable to look for nonlinearity. (There are other methods too).

So, when, if ever, should you bin? I can think of two possible reasons:

  1. The cutoff of your bin is of substantive interest and perhaps part of a hypothesis. For instance, if you are interested in the behavior of people across the lifespan and have hypothesized that it has one pattern below age 18, another from 18-65 and another at older ages, one method would be to bin age in three categories. (Even here, you might use a hockey stick type model, or a spline model with knots set by you).

  2. The model is too "big" to run in a reasonable time. Of course, how big is big depends on your computer and your software and what is "reasonable" is also context specific. if you are making a model that will be run only once, or only once a year, then a running time of many hours might be reasonable. If you need to run it every day, then maybe not.

One other possible reason is that binned models are easier to explain than splines. This is true, but I don't regard it as a good reason. One of your jobs, as a data analyst, is to explain things.

Finally, you might bin because it is demanded of you by your "pointed haired boss" (from Dilbert) who could be a journal editor, a dissertation advisor, a client, a boss and so on.


Whether to bin or not to bin may be answered by the quote (due to George Box?):

All models are wrong, but some models are useful.

Broadly, models are created to either understand the data or to make predictions (and of course for both!).

In your situation I would carry out some experiments and test a range of bin sizes starting with a no bin model.

The "no bin" model could be your baseline model.

For a prediction model your performance metric will help you assess how each bin assignment performed on some holdout data set.

For understanding the model variables you can extract their relative importance following each bin assignment to see if they "make sense" according to accepted theory.

Keep in mind that any time your model restricts your dataset in some way you will likely decrease its information content.

BUT the model may be useful!

  • $\begingroup$ I don't believe, as your post might suggest, that the quote by George Box excuses misdirected modelling decisions just because the result is useful (read: better than nothing). $\endgroup$ – Ryan Volpi Jul 5 '20 at 20:46
  • $\begingroup$ Hi Ryan, Yes and no. We are witness at the moment of models of corona virus spread that are shown to be wrong almost daily but the models are not completely useless. The pattern of spread and the behaviour of the virus is confounding our models but hopefully as more data is collected our models will improve. $\endgroup$ – cousin_pete Jul 6 '20 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.