# Disadvantages of using a regression loss function in multi-class classification

Given $$k > 2$$ classes, consider the following loss function $$\sum_i||y^{(i)} - \hat y^{(i)}||^2$$ Here $$y^{(i)} \in \{0,1\}^k$$ is the $$i^{th}$$ one-hot encoded true label and $$\hat y^{(i)} \in [0,1]^k$$ is the prediction (obtained after applying a sigmoid to the output logits of some neural network).
What are the theoretical and practical disadvantages of using such a loss over something like the cross-entropy loss?

• You might be interested in the comments here. // I made a small edit. If you disagree, please do change it back, but please do explain why you want all of $\mathbb R^k$ to be possible for predictions.
– Dave
Commented Mar 18, 2022 at 10:04
• @Dave thanks for the edit, that was indeed a typo Commented Mar 18, 2022 at 11:32

Squared error used for classification problems is called Brier score and same as log-loss is a strictly proper scoring rule, i.e. it leads to producing well-calibrated probabilities. It is perfectly fine to use squared error as a loss function for classification.

This issue was studied by Hui and Belkin (2020), who conclude:

We argue that there is little compelling empirical or theoretical evidence indicating a clear-cut advantage to the cross-entropy loss. Indeed, in our experiments, performance on nearly all non-vision tasks can be improved, sometimes significantly, by switching to the square loss. Furthermore, training with square loss appears to be less sensitive to the randomness in initialization. We posit that training using the square loss for classification needs to be a part of best practices of modern deep learning on equal footing with cross-entropy.

You may notice in Section 5 of the paper some technical considerations that the authors found to improve training.

• Won't using a squared error loss implicitly imply that we are modelling $y$ as a (multivariate) Gaussian? which as Dave pointed out in his comment is a bad model since it's unbounded Commented Mar 18, 2022 at 13:49
• @helperFunction if you want bounded probabilities, using softmax on the final layer would bound them. On another hand, if you only want to make classifications, you can use the predictions as unbounded scores and use some kind of threshold, without any considerations on the meaning of the scores. Moreover, why would it matter at all what likelihood function does such loss correspond to?
– Tim
Commented Mar 18, 2022 at 13:53
• If I were to "derive" this loss via the likelihood function, then that would be incorrect right? Commented Mar 18, 2022 at 13:59
• @helperFunction it wouldn't correspond to the Bernoulli likelihood anymore, but why would that matter? See also my edit for some more details.
– Tim
Commented Mar 18, 2022 at 14:16

The cross-entropy loss gives you the maximum likelihood estimate (MLE), i.e. if you find the minimum of cross-entropy loss you have found the model (from the family of models you consider) that gives the largest probability to your training data; no other model from your family gives more probability to your training data. (A model family might be e.g. the set of all possible weight assignments to some chosen neural network design.)

Being an MLE helps with mathematical reasoning about the properties of your result because there is lots of theory for MLEs.

Also, cross-entropy would be a little faster to compute than the sum of squared error (SSE) loss you mention.

Some people argue, that SSE loss is inferior because the loss depends not only on the probability of the correct label under your model but also on the distribution of the probabilities that the model gives to the wrong models (since it is not linear).

But, as far as deep neural networks are concerned, the real reason why cross-entropy is used most often (not always) for those, is that experience shows that it is very often leading to better results. We just haven't found something truly better yet (that would also be practical).

• I agree about the MLE part. But even this regression loss can be interpreted as assuming a Gaussian distribution over the sigmoid-ed logits, which might help the modelling in some cases. Can you provide any references of cross-entropy outperforming regression losses by a significant margin in a (multi-class) classification setting? Commented Mar 18, 2022 at 9:16
• @helperFunction How do you figure that a Gaussian distribution, which is unbounded on $\mathbb R$, is a good model for probability values that are limited to $[0,1]?$
– Dave
Commented Mar 18, 2022 at 10:08
• @Dave that's a fair point, I got confused between the ranges of the logits and the ranges of the sigmoid-ed logits Commented Mar 18, 2022 at 11:54
• This sounds like a complex discussion when simply going with the multinomial logistic regression model should suffice. Commented Mar 18, 2022 at 12:38