Regression as classification: advantages?

Question

I have read on many occasions deep learning practitioners recommending to treat regression problems (with continuous variables) as classification problems, by quantizing the output into bins and using cross-entropy loss instead of L2 loss.

For example these notes from Stanfords's CS231n class on CNNs:

It is important to note that the L2 loss is much harder to optimize than a more stable loss such as Softmax. Intuitively, it requires a very fragile and specific property from the network to output exactly one correct value for each input (and its augmentations). Notice that this is not the case with Softmax, where the precise value of each score is less important: It only matters that their magnitudes are appropriate. Additionally, the L2 loss is less robust because outliers can introduce huge gradients. When faced with a regression problem, first consider if it is absolutely inadequate to quantize the output into bins. [...] Classification has the additional benefit that it can give you a distribution over the regression outputs, not just a single output with no indication of its confidence. If you’re certain that classification is not appropriate, use the L2 but be careful: For example, the L2 is more fragile and applying dropout in the network (especially in the layer right before the L2 loss) is not a great idea.

When faced with a regression task, first consider if it is absolutely necessary. Instead, have a strong preference to discretizing your outputs to bins and perform classification over them whenever possible.

Or Pixel-RNN which quantizes pixel values into bins:

Furthermore, in contrast to previous approaches that model the pixels as continuous values [...], we model the pixels as discrete values using a multinomial distribution implemented with a simple softmax layer. We observe that this approach gives both representational and training advantages for our models.

Treating regression as classification to me seems counter-intuitive, since the model cannot differentiate between a point that has been misclassified to a neighboring bin, vs. a distant bin.

Apart from the few points cited above (allowing for small errors, and being more robust to outliers), is there any theoretical evidence – or at least some intuition – as to why treating regression as classification performs better? Is it purely an optimization issue, because the optimization landscape in classification is easier to navigate than in regression? Or are there any other reasons?

I am mostly asking in the context of deep learning, but am also curious about more general answers.

Answer 1

The question gives a number of possible advantages, so I will post possible disadvantages. It is then up to the scientist to evaluate the tradeoff between possible advantages and disadvantages.

1. The reference makes it sound like there are considerable computational advantages. Especially in deep learning settings, the computational issues really do have to be considered. While it is great to be able to prove, mathematically/statistically, that some method is superior to a “trick” in deep learning, if that method cannot be computed in a reasonable amount of time, it is not useful. However, I am not sold on the computational advantages. For instance, if the classifier makes a high-confidence prediction that an observation right near one of the bin boundaries is in the bin on the other side of the boundary (probably only a mild mistake), the entire loss could be dominated by that when you start taking logarithms of small numbers in the cross-entropy loss. This puts the model in a position to get hung up on fixing a mild mistake, perhaps at the expense of making improvements in other areas where the errors are more egregious.

2. The fact that a classifier returns the probability of being in a particular category is appealing. However, neural networks are known to be overconfident in their predictions of these probabilities, and calibrating a multi-class output is not straightforward. Further, techniques exist to estimate conditional distributions, such as quantile estimation.

3. There is not an especially high penalty for bad misses. If the prediction puts high probability on the next bin over from where the observed value is, that incurs the same penalty as putting that same probability in a much higher or lower bin. While this could be argued to give robustness similar to how minimizing absolute loss gives robustness in that large misses are not penalized as severely as they are for square loss (for better or for worse), at least absolute loss penalizes more for large misses than small misses. There is a limit to how much robustness is desired.

(The first and third disadvantages can be combined to say that this approach risks giving large penalties to small misses and small penalties to large misses.)

4. Some of the appeal of this seems to come from classification accuracy being easier to interpret than regression metrics like (root) mean squared error. However, people goof up in interpreting accuracy all the time. I cited a paper on here a few weeks ago (Sundaram & Yermack (2007)) that seemed to be raving about achieving a classification accuracy of $97\%$, despite the majority class making up $97.71\%$ of the observations, meaning that a naïve model could achieve $97.71\%$ classification accuracy (better than their model achieves) by predicting the majority category every time. (This article was published in the top journal in its field (not “a” top journal, “the” top journal), so it is not just the fringe that makes mistakes in evaluating classification accuracy.) Even when the Sundaram & Yermack (2007) classification accuracy scores are above the scores achieved by predicting the majority category every time, the reductions in error rates, which is probably more informative (and can be equivalent to Cohen's kappa), does not scream out, "This model gets an $\text{A}$," the way that a classification accuracy of $97\%$ might. Further, regression metrics like root mean squared error and mean absolute error are in the original units of your measured outcomes, which should have an interpretation by someone who knows the field.

This answer to "Why should binning be avoided at all costs?" is worth a read, even if it is not about the exact same topic. I especially like the last sentence, which I will quote below

My recommendation would be to learn the analytical methods that are applied to the underlying continuous data, and then you will be in a position to determine whether a crude approximation via binning is necessary in a given situation.

REFERENCE

Sundaram, Rangarajan K., and David L. Yermack. "Pay me later: Inside debt and its role in managerial compensation." The Journal of Finance 62.4 (2007): 1551-1588.