11
$\begingroup$

Does cross-entropy cost make sense in the context of regression? (as opposed to classification) If so, could you give a toy example through tensorflow and if not, why not?

I was reading about cross entropy in Neural Networks and Deep Learning by Michael Nielsen and it seems like something that could naturally be used for regression as well as classification, but I don't understand how you'd apply it efficiently in tensorflow since the loss functions take logits (which I don't really understand either) and they're listed under Classification here

$\endgroup$
  • 1
    $\begingroup$ I found here on quora that states different from what is accepted as an answer for this question $\endgroup$ – Siddharth Shakya Jul 25 '18 at 7:50
  • $\begingroup$ If you read the whole response, you see that he gives a "continuous version" of cross-entropy which is pretty cool, but it turns out to just be the Mean Squared Error (MSE). $\endgroup$ – JacKeown Jul 26 '18 at 13:44
10
$\begingroup$

No, it doesn't make sense to use cross-entropy for regression using the Tensorflow functions that you mentioned. In Tensorflow, “cross-entropy” is shorthand (or jargon) for “categorical cross entropy.” Categorical cross entropy is an operation on probabilities. In the binary classification case, cross entropy is defined as $$ H=-\frac{1}{N}\sum_{i=1}^N y_i\log(\hat{p_i})+(1-y_i)\log(1-\hat{p_i}) $$

where $\hat{p_i}$ is the estimated probability that $y_i=1$. The expression for $H$ can be readily generalized to more than 2 outcomes; it just requires breaking the $p_i$ out into each outcome, and incrementing by the log of the $p_i$ corresponding to the true outcome (see: Machine Learning: Should I use a categorical cross entropy or binary cross entropy loss for binary predictions?). Note that because $y_i\in\{0,1\}$, the expression effectively has a "on/off switch" because the coefficient to the log will be nonzero for exactly one term of the sum. Note also that this expression is just a scalar multiple of the log-likelihood of a Bernoulli model of the data, so this has some connection to the classical MLE procedures.

A neural net trained to classify objects is obviously applicable in this case. But in a regression context, the target $y$ can be any real number. Moreover, you're not estimating probabilities of class membership, but just a real number. So, for example, $-1.6$ would be a perfectly valid regression prediction, even though it's clearly not a probability. Moreover, taking the log of $-1.6$ does not return a real number!

The cross entropy with logits function is just for convenience and stability. Obviously one can define a categorical cross entropy loss that does not require logits (probits, cloglog). Logits are used all over statistics and machine learning. I feel that this portion of your question has been answered here and here.

Some of the answers and comments in this thread suggest that this answer is wrong because one can also define cross-entropy losses which correspond to real-valued outcomes instead of categorical outcomes. These comments are correct only insofar as "cross entropy loss" is a general family of loss functions; however, these comments are not correct in the specific sense of this question because this is not what the TensorFlow authors mean when naming functions like tf.nn.sigmoid_cross_entropy_with_logits. Perhaps TensorFlow authors picked an ambiguous or misleading name for this family of functions (each person can make up their mind about this); however, that does not change the fact that the function computes a specific loss which is intimately related to this expression for $H$.

$\endgroup$
  • 4
    $\begingroup$ Although, it should be mentioned that using binary crossentropy as the loss function in a regression task where the output values are real values in the range [0,1] is a pretty reasonable and valid thing to do. $\endgroup$ – today Nov 21 '18 at 8:45
  • 1
    $\begingroup$ This answer is wrong. For cross entropy you need to establish a distribution. For regression you may of course set a Gaussian and compute the cross entropy which in this case will give the same weights as simple squared loss. Or you may pick a heavy tailed distribution. $\endgroup$ – Cagdas Ozgenc Nov 21 '18 at 14:45
  • $\begingroup$ @CowboyTrader This seems like a nitpick; “cross entropy” is a common shorthand for “categorical cross entropy.” The question is concerned with why cross entropy functions in software such as Tensorflow assumes categorical targets, so answering the question in these terms is absolutely correct. $\endgroup$ – Reinstate Monica Nov 21 '18 at 15:08
  • $\begingroup$ There may be different point of views. But the fact that Tensorflow or some other API defines something incorrectly is not a justification for that. For example logit is not the only function you can use with cross entropy for a categorical output. All functions with [0,1] range can be used as a link function. Now are we going to say that the definition of cross entropy is only for logistic function? $\endgroup$ – Cagdas Ozgenc Nov 21 '18 at 15:19
  • $\begingroup$ You’re arguing against a position that no one has taken. This is a question about the jargon used in tensorflow. The question is expressly not about general definitions of cross entropy, and we know this because it asks about Tensorflow in the title and body. Your point about cross entropy with logits misunderstands the purpose of the function: it provides convenience and stability in the case of a model that uses logits; models that use cross entropy and logits are common in deep learning. No one is suggesting that Tensorflow authors are attempting to define math concepts. $\endgroup$ – Reinstate Monica Nov 21 '18 at 15:40
4
$\begingroup$

The answer given by @Sycorax is correct. However, it is worth mentioning that using (binary) cross-entropy in a regression task where the output values are in the range [0,1] is a valid and reasonable thing to do. Actually, it is used in image autoencoders (e.g. here and this paper). You might be interested to see a simple mathematical proof of why it works in this case in this answer.

$\endgroup$
  • $\begingroup$ Loss functions can viewed as likelihoods / posteriors or some monotonic transformation of them. So, while it is true that in some regression models a loss similar to the cross-entropy might make sense, it might not be a reasonable approach to deal with any regression where the outputs are in a $[0, 1]$ range. $\endgroup$ – InfProbSciX Nov 21 '18 at 14:36
  • $\begingroup$ @InfProbSciX "it might not be a reasonable approach to deal with any regression where the outputs are in a [0,1] range." So "reasonable" in what sense? Or how do you define the reasonability of loss function for a specific task? I suspect that statement might be true for any loss function. Is there any loss function that would be reasonable to use for all kinds of regression tasks, of course after defining the "reasonable"? $\endgroup$ – today Nov 21 '18 at 14:41
  • $\begingroup$ The way I'd define reasonable is by constructing a model law. For example, in a regression framework such as $Y = f_{\theta}(X) + \epsilon$ where $\epsilon$ are i.i.d. errors - say normally distributed, the negative log-likelihood is exactly the squared loss. In a setting where the model law looks like $Y \sim Bernoulli(p_{\theta})$, the negative log-likelihood is exactly the binary cross entropy. Where the law is a linear regression with a normal prior on the coefs, the loss corresponds to the L2 penalty and so on. Where possible, I'd construct a law and then derive a loss. $\endgroup$ – InfProbSciX Nov 21 '18 at 14:46
  • $\begingroup$ @InfProbSciX Thanks for your reply. So as you mentioned, depending on the regression task (and the assumptions on the distribution of data, errors, etc.) a loss function might not be reasonable to be used. And, as I mentioned, this is true for all loss functions, including crossentropy. Of course, I see your point that just because the output values are in the range [0,1] does not guarantee that crossentropy is the optimal choice loss function and I was not trying to convey the otherwise in my answer. $\endgroup$ – today Nov 21 '18 at 15:14
1
$\begingroup$

Deep learning frameworks often mix models and losses and refer to the cross-entropy of a multinomial model with softmax nonlinearity by cross_entropy, which is misleading. In general, you can define cross-entropy for arbitrary models.

For a Gaussian model with varying mean but fixed diagonal covariance, it is equivalent to MSE. For a general covariance, cross-entropy would correspond to a squared Mahalanobis distance. For an exponential distribution, the cross-entropy loss would look like $$f_\theta(x) y - \log f_\theta(x),$$ where $y$ is continuous but non-negative. So yes, cross-entropy can be used for regression.

$\endgroup$
1
$\begingroup$

Unfortunately, the as of now accepted answer by @Sycorax, while detailed, is incorrect.

Actually, a prime example of regression through categorical cross-entropy -- Wavenet -- has been implemented in TensorFlow.

The principle is that you discretize your output space and then your model only predicts the respective bin; see Section 2.2 of the paper for an example in the sound modelling domain. So while technically the model performs classification, the eventual task solved is regression.

An obvious downside is, that you lose output resolution. However, this may not be a problem (at least I think that the Google's artificial assistant spoke a very humanly voice) or you can play around with some post-processing, e.g. interpolating between the most probable bin and it's two neighbours.

On the other hand, this approach makes the model much more powerful compared to the usual single-linear-unit output, i.e. allowing to express multi-modal predictions or to assess it's confidence. Note though that the latter can be naturally achieved by other means, e.g. by having an explicit (log)variance output as in Variational Autoencoders.

Anyway, this approach does not scale well to more-dimensional output, because then the size of the output layer grows exponentially, making it both computational and modelling issue..

$\endgroup$
  • 1
    $\begingroup$ I see what you're saying, but I wouldn't personally consider discretizing your output space as performing "regression" as much as it is approximating a regression problem using classification...but I guess it's just a matter of terminology/convention. $\endgroup$ – JacKeown Nov 27 '18 at 19:09
  • 1
    $\begingroup$ Agreed. The 32-bit float space is discrete anyway :-) $\endgroup$ – dedObed Nov 27 '18 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.