# Tensorflow Cross Entropy for Regression?

Does cross-entropy cost make sense in the context of regression? (as opposed to classification) If so, could you give a toy example through tensorflow and if not, why not?

I was reading about cross entropy in Neural Networks and Deep Learning by Michael Nielsen and it seems like something that could naturally be used for regression as well as classification, but I don't understand how you'd apply it efficiently in tensorflow since the loss functions take logits (which I don't really understand either) and they're listed under Classification here

• I found here on quora that states different from what is accepted as an answer for this question Jul 25 '18 at 7:50
• If you read the whole response, you see that he gives a "continuous version" of cross-entropy which is pretty cool, but it turns out to just be the Mean Squared Error (MSE). Jul 26 '18 at 13:44
• the link to tensorflow website is broken
– Long
Apr 29 '20 at 5:27
• @Long, just fixed the link. Apr 30 '20 at 4:20
• @JacKeown it's not fixed yet.
– Long
Apr 30 '20 at 6:28

No, it doesn't make sense to use TensorFlow functions like tf.nn.sigmoid_cross_entropy_with_logits for a regression task. In TensorFlow, “cross-entropy” is shorthand (or jargon) for “categorical cross entropy.” Categorical cross entropy is an operation on probabilities. A regression problem attempts to predict continuous outcomes, rather than classifications.

The jargon "cross-entropy" is a little misleading, because there are any number of cross-entropy loss functions; however, it's a convention in machine learning to refer to this particular loss as "cross-entropy" loss.

If we look beyond the TensorFlow functions that you link to, then of course there are any number of possible cross-entropy functions. This is because the general concept of cross-entropy is about the comparison of two probability distributions. Depending on which two probability distributions you wish to compare, you may arrive at a different loss than the typical categorical cross-entropy loss. For example, the cross-entropy of a Gaussian target with some varying mean but fixed diagonal covariance reduces to mean-squared error. The general concept of cross-entropy is outlined in more detail in these questions:

• Although, it should be mentioned that using binary crossentropy as the loss function in a regression task where the output values are real values in the range [0,1] is a pretty reasonable and valid thing to do. Nov 21 '18 at 8:45
• @today I think this is a practice that some people adopt for practical reasons (e.g. the neural network converges more quickly), but I'm not sure that this model has any relationship to the comparison of two probability distributions. Perhaps you could show that there is a relationship between a continuously-valued target in $[0,1]$ and binary cross-entropy?
– Sycorax
Jan 21 '20 at 2:30

The answer given by @Sycorax is correct. However, it is worth mentioning that using (binary) cross-entropy in a regression task where the output values are in the range [0,1] is a valid and reasonable thing to do. Actually, it is used in image autoencoders (e.g. here and this paper). You might be interested to see a simple mathematical proof of why it works in this case in this answer.

• Loss functions can viewed as likelihoods / posteriors or some monotonic transformation of them. So, while it is true that in some regression models a loss similar to the cross-entropy might make sense, it might not be a reasonable approach to deal with any regression where the outputs are in a $[0, 1]$ range. Nov 21 '18 at 14:36
• @InfProbSciX "it might not be a reasonable approach to deal with any regression where the outputs are in a [0,1] range." So "reasonable" in what sense? Or how do you define the reasonability of loss function for a specific task? I suspect that statement might be true for any loss function. Is there any loss function that would be reasonable to use for all kinds of regression tasks, of course after defining the "reasonable"? Nov 21 '18 at 14:41
• The way I'd define reasonable is by constructing a model law. For example, in a regression framework such as $Y = f_{\theta}(X) + \epsilon$ where $\epsilon$ are i.i.d. errors - say normally distributed, the negative log-likelihood is exactly the squared loss. In a setting where the model law looks like $Y \sim Bernoulli(p_{\theta})$, the negative log-likelihood is exactly the binary cross entropy. Where the law is a linear regression with a normal prior on the coefs, the loss corresponds to the L2 penalty and so on. Where possible, I'd construct a law and then derive a loss. Nov 21 '18 at 14:46
• @InfProbSciX Thanks for your reply. So as you mentioned, depending on the regression task (and the assumptions on the distribution of data, errors, etc.) a loss function might not be reasonable to be used. And, as I mentioned, this is true for all loss functions, including crossentropy. Of course, I see your point that just because the output values are in the range [0,1] does not guarantee that crossentropy is the optimal choice loss function and I was not trying to convey the otherwise in my answer. Nov 21 '18 at 15:14
• This need not be restricted to output range of [0,1] only. Users of deep models prefer cross entropy over MSE. I have seen non [0,1] regression output being compressed to [0,1] using a sigmoid just to use cross entropy loss function after that. Of course later on we need to apply a logit function later on the predicted values to get back the original range. Dropout, BatchNorm etc are more stable with Cross Entropy loss and in general the convergence is also faster Aug 16 at 5:50

Deep learning frameworks often mix models and losses and refer to the cross-entropy of a multinomial model with softmax nonlinearity by cross_entropy, which is misleading. In general, you can define cross-entropy for arbitrary models.

For a Gaussian model with varying mean but fixed diagonal covariance, it is equivalent to MSE. For a general covariance, cross-entropy would correspond to a squared Mahalanobis distance. For an exponential distribution, the cross-entropy loss would look like $$f_\theta(x) y - \log f_\theta(x),$$ where $$y$$ is continuous but non-negative. So yes, cross-entropy can be used for regression.

Unfortunately, the as of now accepted answer by @Sycorax, while detailed, is incorrect.

Actually, a prime example of regression through categorical cross-entropy -- Wavenet -- has been implemented in TensorFlow.

The principle is that you discretize your output space and then your model only predicts the respective bin; see Section 2.2 of the paper for an example in the sound modelling domain. So while technically the model performs classification, the eventual task solved is regression.

An obvious downside is, that you lose output resolution. However, this may not be a problem (at least I think that the Google's artificial assistant spoke a very humanly voice) or you can play around with some post-processing, e.g. interpolating between the most probable bin and it's two neighbours.

On the other hand, this approach makes the model much more powerful compared to the usual single-linear-unit output, i.e. allowing to express multi-modal predictions or to assess it's confidence. Note though that the latter can be naturally achieved by other means, e.g. by having an explicit (log)variance output as in Variational Autoencoders.

Anyway, this approach does not scale well to more-dimensional output, because then the size of the output layer grows exponentially, making it both computational and modelling issue..

• I see what you're saying, but I wouldn't personally consider discretizing your output space as performing "regression" as much as it is approximating a regression problem using classification...but I guess it's just a matter of terminology/convention. Nov 27 '18 at 19:09
• Agreed. The 32-bit float space is discrete anyway :-) Nov 27 '18 at 20:46

I've revisited this question as I now disagree with the answer I previously accepted. Cross entropy loss CAN be used in regression (although it isn't common.)

It comes down to the fact that cross-entropy is a concept that only makes sense when comparing two probability distributions. You could consider a neural network which outputs a mean and standard deviation for a normal distribution as its prediction. It would then be punished more harshly for being more confident about bad predictions. So yes, it makes sense, but only if you're outputting a distribution in some sense. The link from @SiddharthShakya in a comment to my original question shows this.

– Sycorax
Jan 19 '20 at 23:11
• I understand your objection, but I plan on leaving the question as is because it represents my original query which I feel could help people with the same question I had. At any rate, the entire post should contain enough information overall. Jan 21 '20 at 0:26

Yes, sure.

What is Cross-Entropy?

Let's think about what is Cross-Entropy (CE). CE cost in the context of PyTorch or another Frameworks can mean a different thing compare to MATH. Originally cross-entropy is some form of KL-divergence between distributions: https://sites.google.com/site/burlachenkok/articles/properties-of-kl-divergence

Reasons:

1. In the context of Machine Learning, the very often first argument for CE is typically vector from probability simplex such that it has one component equal to one. So input for CE is probability mass function (p.m.f.) in a discrete case.

2. Inside PyTorch let's say there is an extra transformation called in STATS as "symmetric logistics transform" which $$\dfrac{exp(f_i(x)}{exp(f_1(x))+exp(f_2(x))+\dots + exp(f_k(x))}$$ What it's interesting it's a bijective mapping from Euclidane space into probabilistic simplex.

Now, what is a regression?

Regression in the context of probability theory means $$E_{z}[y(x,z)|x]$$ where by z I denote unobserved variables. In the context of machine learning, it means any predictor with an output single scalar variable or multiple scalar variables and it is not necessary conditional expectation.

So as you see - there is already too much confusion with various terminology and basic terms in STATS and ML.

If your model provides K scalar outputs (called sometimes logits) it can be plugged into CE with the symmetric logistic transformation. For the record - logits are just unbound scores from $$\mathbb{R}$$ such that class with maximum score is your prediction.

I think the answer you can do whatever you want, and people do crazy things with Loss in STATS, Optimization, and Deep Learning Applications.

I can not give you an exact answer because CE very often raised with classification models e.g. in Deep Learning or with Decision Trees. But if your question because you design such a system - it's better to allow more expressive power in Loss construction.