13
$\begingroup$

Does the cross-entropy cost make sense in the context of regression (as opposed to classification)? If so, could you give a toy example through TensorFlow? If not, why not?

I was reading about cross-entropy in Neural Networks and Deep Learning by Michael Nielsen and it seems like something that could naturally be used for regression as well as classification, but I don't understand how you'd apply it efficiently in TensorFlow since the loss functions take logits (which I don't really understand either) and they're listed under classification here

$\endgroup$
  • 2
    $\begingroup$ I found here on quora that states different from what is accepted as an answer for this question $\endgroup$ – Siddharth Shakya Jul 25 '18 at 7:50
  • $\begingroup$ If you read the whole response, you see that he gives a "continuous version" of cross-entropy which is pretty cool, but it turns out to just be the Mean Squared Error (MSE). $\endgroup$ – JacKeown Jul 26 '18 at 13:44
11
$\begingroup$

No, it doesn't make sense to use TensorFlow functions like tf.nn.sigmoid_cross_entropy_with_logits for a regression task. In TensorFlow, “cross-entropy” is shorthand (or jargon) for “categorical cross entropy.” Categorical cross entropy is an operation on probabilities. A regression problem attempts to predict continuous outcomes, rather than classifications.

The jargon "cross-entropy" is a little misleading, because there are any number of cross-entropy loss functions; however, it's a convention in machine learning to refer to this particular loss as "cross-entropy" loss.

If we look beyond the TensorFlow functions that you link to, then of course there are any number of possible cross-entropy functions. This is because the general concept of cross-entropy is about the comparison of two probability distributions. Depending on which two probability distributions you wish to compare, you may arrive at a different loss than the typical categorical cross-entropy loss. For example, the cross-entropy of a Gaussian target with some varying mean but fixed diagonal covariance reduces to mean-squared error. The general concept of cross-entropy is outlined in more detail in these questions:

$\endgroup$
  • 5
    $\begingroup$ Although, it should be mentioned that using binary crossentropy as the loss function in a regression task where the output values are real values in the range [0,1] is a pretty reasonable and valid thing to do. $\endgroup$ – today Nov 21 '18 at 8:45
  • $\begingroup$ @today I think this is a practice that some people adopt for practical reasons (e.g. the neural network converges more quickly), but I'm not sure that this model has any relationship to the comparison of two probability distributions. Perhaps you could show that there is a relationship between a continuously-valued target in $[0,1]$ and binary cross-entropy? $\endgroup$ – Sycorax says Reinstate Monica Jan 21 at 2:30
4
$\begingroup$

The answer given by @Sycorax is correct. However, it is worth mentioning that using (binary) cross-entropy in a regression task where the output values are in the range [0,1] is a valid and reasonable thing to do. Actually, it is used in image autoencoders (e.g. here and this paper). You might be interested to see a simple mathematical proof of why it works in this case in this answer.

$\endgroup$
  • $\begingroup$ Loss functions can viewed as likelihoods / posteriors or some monotonic transformation of them. So, while it is true that in some regression models a loss similar to the cross-entropy might make sense, it might not be a reasonable approach to deal with any regression where the outputs are in a $[0, 1]$ range. $\endgroup$ – InfProbSciX Nov 21 '18 at 14:36
  • $\begingroup$ @InfProbSciX "it might not be a reasonable approach to deal with any regression where the outputs are in a [0,1] range." So "reasonable" in what sense? Or how do you define the reasonability of loss function for a specific task? I suspect that statement might be true for any loss function. Is there any loss function that would be reasonable to use for all kinds of regression tasks, of course after defining the "reasonable"? $\endgroup$ – today Nov 21 '18 at 14:41
  • $\begingroup$ The way I'd define reasonable is by constructing a model law. For example, in a regression framework such as $Y = f_{\theta}(X) + \epsilon$ where $\epsilon$ are i.i.d. errors - say normally distributed, the negative log-likelihood is exactly the squared loss. In a setting where the model law looks like $Y \sim Bernoulli(p_{\theta})$, the negative log-likelihood is exactly the binary cross entropy. Where the law is a linear regression with a normal prior on the coefs, the loss corresponds to the L2 penalty and so on. Where possible, I'd construct a law and then derive a loss. $\endgroup$ – InfProbSciX Nov 21 '18 at 14:46
  • $\begingroup$ @InfProbSciX Thanks for your reply. So as you mentioned, depending on the regression task (and the assumptions on the distribution of data, errors, etc.) a loss function might not be reasonable to be used. And, as I mentioned, this is true for all loss functions, including crossentropy. Of course, I see your point that just because the output values are in the range [0,1] does not guarantee that crossentropy is the optimal choice loss function and I was not trying to convey the otherwise in my answer. $\endgroup$ – today Nov 21 '18 at 15:14
4
$\begingroup$

Deep learning frameworks often mix models and losses and refer to the cross-entropy of a multinomial model with softmax nonlinearity by cross_entropy, which is misleading. In general, you can define cross-entropy for arbitrary models.

For a Gaussian model with varying mean but fixed diagonal covariance, it is equivalent to MSE. For a general covariance, cross-entropy would correspond to a squared Mahalanobis distance. For an exponential distribution, the cross-entropy loss would look like $$f_\theta(x) y - \log f_\theta(x),$$ where $y$ is continuous but non-negative. So yes, cross-entropy can be used for regression.

$\endgroup$
2
$\begingroup$

Unfortunately, the as of now accepted answer by @Sycorax, while detailed, is incorrect.

Actually, a prime example of regression through categorical cross-entropy -- Wavenet -- has been implemented in TensorFlow.

The principle is that you discretize your output space and then your model only predicts the respective bin; see Section 2.2 of the paper for an example in the sound modelling domain. So while technically the model performs classification, the eventual task solved is regression.

An obvious downside is, that you lose output resolution. However, this may not be a problem (at least I think that the Google's artificial assistant spoke a very humanly voice) or you can play around with some post-processing, e.g. interpolating between the most probable bin and it's two neighbours.

On the other hand, this approach makes the model much more powerful compared to the usual single-linear-unit output, i.e. allowing to express multi-modal predictions or to assess it's confidence. Note though that the latter can be naturally achieved by other means, e.g. by having an explicit (log)variance output as in Variational Autoencoders.

Anyway, this approach does not scale well to more-dimensional output, because then the size of the output layer grows exponentially, making it both computational and modelling issue..

$\endgroup$
  • 1
    $\begingroup$ I see what you're saying, but I wouldn't personally consider discretizing your output space as performing "regression" as much as it is approximating a regression problem using classification...but I guess it's just a matter of terminology/convention. $\endgroup$ – JacKeown Nov 27 '18 at 19:09
  • 1
    $\begingroup$ Agreed. The 32-bit float space is discrete anyway :-) $\endgroup$ – dedObed Nov 27 '18 at 20:46
-1
$\begingroup$

I've revisited this question as I now disagree with the answer I previously accepted. Cross entropy loss CAN be used in regression (although it isn't common.)

It comes down to the fact that cross-entropy is a concept that only makes sense when comparing two probability distributions. You could consider a neural network which outputs a mean and standard deviation for a normal distribution as its prediction. It would then be punished more harshly for being more confident about bad predictions. So yes, it makes sense, but only if you're outputting a distribution in some sense. The link from @SiddharthShakya in a comment to my original question shows this.

$\endgroup$
  • 1
    $\begingroup$ This answer seems to answer the question in a different way than it's asked. The functions that you linked to in the question are about a specific kind of cross-entropy loss, and your question seems to ask if those functions can be used in regression, and my answer is written as if you are asking how to use those functions you link to. The answer here seems to answer the question "Can cross-entropy be generalized beyond classification?" Editing the Q would make it clear that the focus is on how mathematical concepts are defined, rather than focusing on how to use Tensorflow functions. $\endgroup$ – Sycorax says Reinstate Monica Jan 19 at 23:11
  • $\begingroup$ I understand your objection, but I plan on leaving the question as is because it represents my original query which I feel could help people with the same question I had. At any rate, the entire post should contain enough information overall. $\endgroup$ – JacKeown Jan 21 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.