8
$\begingroup$

I am an MBA Student taking courses in statistics. I was doing some reading online and came across Carstensen (2020), which I unfortunately do not have access to.

This article suggests that "categorizing quantitative variable" is bad - but I am not sure why this is. Our professor in our statistics courses showed us examples where a decision tree was fit to some data where the dependent variable was originally continuous, but then "binned" into groups. In his example, this strategy seemed to work fine - but some other references are suggesting that this is bad practice.

Can someone please explain why doing this might not be ideal? I understand that if you make arbitrary bins that this could lead to problems and lose information in the process - but if the bins are made carefully and rigorously tested, why is it bad to "categorize quantitative variables"?

$\endgroup$
12
  • 10
    $\begingroup$ There's several posts on site already that discuss this issue in some detail. $\endgroup$
    – Glen_b
    Commented Oct 14, 2022 at 5:20
  • 2
    $\begingroup$ Note that the linked source is discussing dichotomization of independent predictor variables - this allows you to tune a continuous decision threshold to optimize some quantity. The discretization of the final continuous output is often a necessary step when the goal is to make a decision, which ultimately is a selection among discrete outputs by its nature. That said, you typically want to do this discretization as late as possible in the analysis. $\endgroup$ Commented Oct 14, 2022 at 13:20
  • $\begingroup$ @NuclearHoagie Is right that a discrete decision might be necessary, but an interesting property of “classification” problems is that there can be more decisions than categories! The continuum of predictions helps make that decision, as is discussed in the link. $\endgroup$
    – Dave
    Commented Oct 14, 2022 at 13:27
  • 3
    $\begingroup$ The entirety of the book chapter you refer to is a direct quote of Problems Caused by Categorizing Continuous Variables which contains further references. This topic has been discussed many times on CV; see for example 1, 2, 3. $\endgroup$
    – dipetkov
    Commented Oct 14, 2022 at 21:53
  • 1
    $\begingroup$ @dipetkov While you are entitled to your opinion & preferences, that's not really how this site works. Any person can write an answer to an open question at any time. Duplicate threads aren't great because the same material is repeated again and again; this goes against the goal of creating single, canonical answers for questions. Moreover, closing questions as duplicates helps future users by improving the SEO of the answers and helping users find when the same question is asked in different ways. More information about the Stack Exchange philosophy is explained in the help center. $\endgroup$
    – Sycorax
    Commented Oct 16, 2022 at 21:14

3 Answers 3

21
$\begingroup$

There are two main issues here that militate against discretisation into bins for statistical analysis. The first is that this generally involves some loss of information, since different values are put into the same bin, and so there is a corresponding loss of statistical power in the analysis. The second is that such an approach fails to produce inferences that change in a continuous manner with the continuous variable of interest; instead the inferences change over the bins and so they appear as "jumps" when looked at on the original continuous scale. Neither of these problems are necessarily fatal, particularly if you have a reasonably large number of bins to reduce the loss of information.

Ideally, we would be able to model continuous data with models that treat them as continuous. However, continuous models generally involve some specification of a parametric family of distributions that limits the generality of analysis, and this means that there are distributional assumptions that might or might not fit the data well. (By contrast, in the discrete case the multinomial distribution fits all possible probability distributions for the number of outcomes occurring over a finite, or even countably infinite, set of finite bins.) It is usually possible to fit a continuous model that works well on the data if you have an expansive set of tools and the ability to generalise continuous models when they don't fit well. The difficulty that some analysts encounter is that they may find that standard continuous models they use for inference (e.g., linear regression) aren't fitting their continuous variables well and they may reach the limit of their ability to generalise these models effectively to improve things. In such cases, analysts sometimes fall back on the discretisation into bins in order to allow them to apply highly general discrete models (e.g., the multinomial) that do not assume any particular distributional forms. This is a trade-off --- you lose some information and the ability to make smooth inferences about a continuous variable, but you have a high level of generality of the remaining discretised variable and it falls within well-known model forms that are easy to apply.

I note your proposal that discretisation might be okay "...if the bins are made carefully and rigorously tested...". It is easy to say that, but how do you propose to test them (against what), if not by comparing them to an initial analysis of the continuous variable? If you "rigorously test" the discretised method and inference by comparing it to a continuous model that is taken to be the correct analysis, then you already presumably have a well-fitted continuous model, so what is the point of the discretisation? If you can't get that comparator, then what exactly is your proposed "rigorous test" doing? It is not necessarily impossible to answer these questions sensibly, but you would need to think more about what exactly you propose to do and how it helps you.

The above gives you a practical idea of the issue, but I would be remiss if I did not expand on this with a little excursion into the philosophy and foundations of mathematics and computing. In particular, it is worth noting that all continuous data is discretised to some extent in statistical analysis, due to the finite precision of computational representation of numbers. At best we represent continuous variables up to some finite level of precision (e.g., standard precision under floating-point arithmetic) and so we always implicitly use a discretised scale where the "bins" are tiny intervals that are too small to be differentiated under the computational representation. (Usually these bins are precise enough to avoid duplicate values of the "continuous" variable.) Since this is a necessity of analysis, there cannot be any in principle objection to some discretisation of continuous variables in analysis, and so the question becomes one of degree. To take a deeper excursion into the philosophy of mathematics, finitists like Doron Zeilberger would go further with this representation argument and object even to the assertion that continuous random variables, continuous functions, or infinite sets exist; they would say that all purportedly continuous variables are actually discrete, up to the finite level of accuracy of the computational representation, and so the real question is only whether we want to aggregate smaller bins into larger bins.

$\endgroup$
8
  • 2
    $\begingroup$ Thank you! I like your philosophical argument - I used to work in an engineering lab where we recorded how well different machines/appliances could measure things like pressure and temperature. More expensive machines could reliably and consistently measure up to a hundredth of a degree - whereas cheaper machines could only do this to a tenth of a degree. But even 31.221 degrees Celsius .... on some level, perhaps this is still fundamentally a discrete measurement? $\endgroup$
    – stats_noob
    Commented Oct 14, 2022 at 5:01
  • 3
    $\begingroup$ Comments are for clarification of issues related to the post, not for pinging people to help you on your other questions. $\endgroup$
    – Glen_b
    Commented Oct 14, 2022 at 5:21
  • 1
    $\begingroup$ +1 especially for mentioning finitist approach. $\endgroup$ Commented Oct 14, 2022 at 7:23
  • 3
    $\begingroup$ Along the lines of Zeilberger's argument, one might even go so far as to say that the very nature of reality itself is discrete (e.g., Planck length, time), and that the notion of continuity is actually an abstraction that we impose on reality that affords us the conveniences that a continuous mathematical model provides. $\endgroup$
    – heropup
    Commented Oct 14, 2022 at 18:53
  • 3
    $\begingroup$ @Ben Aggregation bias is worse than "jumps": with discretization you can both obscure actual relationships, and produce inferences that functional relationships exist which are simply artifacts of the particular choice of discretization (number of bins, and which observations belong to which bin), but otherwise have no basis in the relationship between the variables . $\endgroup$
    – Alexis
    Commented Oct 15, 2022 at 0:37
13
$\begingroup$

Others answers have discussed how discretization throws away information, which can hide effects which would be discovered if the continuous data were used. But sometimes the loss of information actually creates illusory effects in the data!

For example, suppose I'm trying to determine if a particular Northern species of bird migrates south in the winter. In the first week of November, I tag each bird caught and released from a research site at, say, $47.5^\circ$ North latitude. To get a larger sample size, I also tag the birds from a more northern site, at $51^\circ$ North latitude. Then I use the tags to locate the birds in December, seeing if they trend south as it gets colder.

Suppose that in reality, the birds don't migrate at all, but just mill about, some heading south, some north. The continuous data would reveal that there is no migratory trend whatsoever, only random displacements between November and December. But I decide (foolishly) to discretize my data into bins $5^\circ$ across. None of the birds I tagged made it below $40^\circ$ or above $60^\circ$, so I break my sample range into four bins, $40^\circ$-$45^\circ$, $45^\circ$-$50^\circ$, $50^\circ$-$55^\circ$, and $55^\circ$-$60^\circ$. This feels like a natural discretization to me, as the ranges have nice, round-number endpoints.

The first site is in the center of a range, so it doesn't cause me any problems. Most of the birds tagged at the $47.5^\circ$ site stay in the $45^\circ$-$50^\circ$ bin, but some of the outliers end up in both the $40^\circ$-$45^\circ$ and $50^\circ$-$55^\circ$ bins. However, the $51^\circ$ site is close to the bottom of its bin, so birds tagged there are much more likely to randomly move to the bin below than the bin above. I notice this pattern in my data, and wrongly conclude that the northern population of birds tend to travel south during November.

When I publish my results in an esteemed ornithology journal, I don't include the latitudes at which the birds were tagged, only the bin in which they were tagged. I do include statistical tests, including a $p$-value below 0.05, which supports the claim that the birds trend into southern bins during Novemeber. My readers have no way of knowing that the reason for this trend is that my discretization biased my methodology, and are convinced that I have discovered a real phenomenon.

$\endgroup$
1
  • 1
    $\begingroup$ Pedantic side note: I just realized that a random walk starting in the northern hemisphere will actually trend slightly south on average because of the curvature of the Earth, so if my sample sizes were large enough I might actually notice this effect! My point about illusory effects still stands, though. $\endgroup$ Commented Oct 15, 2022 at 2:38
5
$\begingroup$

A different issue is that the categorical model may be unnecessarily hard to fit, caused by the philosophical mismatch of the model to the problem.

Suppose I have a large spring. The farther I pull it, $x$, the more force it exerts, $y$. This is an extremely straightforward continuous relationship -- pull a bit farther, feel a bit more force. I could fit a model $y = ax$ with just one parameter, $a$. If Hooke's law applies, then just a couple data points will give me a very accurate estimate of $a$.

Now consider the categorical approach. I could break distances up into one hundred buckets, $x \in [0,1]$, $x \in [1,2]$, ..., $x \in [99,100]$. Now I have a model with at least one hundred parameters. It is capable of modeling many relationships that I know are impossible, such as the force going up and back down and back up etc. as the spring is stretched farther and farther. I may need hundreds or thousands of data points to fit the model. And it still won't predict very accurately at resolutions below $1$.

To fix this, I could just use a couple buckets, $x \in [0,50]$ and $x \in [50,100]$. Now I don't need as much data and my model is simple, but very inaccurate.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.