Tensorflow Cross Entropy for Regression?

Question

Does cross-entropy cost make sense in the context of regression? (as opposed to classification) If so, could you give a toy example through tensorflow and if not, why not?

I was reading about cross entropy in Neural Networks and Deep Learning by Michael Nielsen and it seems like something that could naturally be used for regression as well as classification, but I don't understand how you'd apply it efficiently in tensorflow since the loss functions take logits (which I don't really understand either) and they're listed under Classification here

Answer 1

In general, it doesn't make sense to use TensorFlow functions like tf.nn.sigmoid_cross_entropy_with_logits for a regression task. In TensorFlow, “cross-entropy” is shorthand (or jargon) for “categorical cross entropy.”

The jargon "cross-entropy" is a little misleading, because there are any number of cross-entropy loss functions; however, it's a convention in machine learning to refer to this particular loss as "cross-entropy" loss.

If we look beyond the TensorFlow functions that you link to, then of course there are any number of possible cross-entropy functions. This is because the general concept of cross-entropy is about the comparison of two probability distributions. Depending on which two probability distributions you wish to compare, you may arrive at a different loss than the typical categorical cross-entropy loss. For example, the cross-entropy of a Gaussian target with some varying mean but fixed diagonal covariance reduces to mean-squared error. The general concept of cross-entropy is outlined in more detail in these questions:

• Do neural networks learn a function or a probability density function?

• How to construct a cross-entropy loss for general regression targets?

• Neural Network Classification - targetting class probability and not the class themselves

---

The kind of "regression" suggested in the question supposes that we can replace the binary $y$ labels with any proportions. This section shows that this substitution is incorrect in general.

In TensorFlow, the cross-entropy loss function is the negative log-likelihood of the binomial distribution. For a single sample, the loss is given by $$ L = -y \log f(x) - (1-y) \log (1-f(x)) $$ where $y$ is the binary lable, $f$ is the model and $x$ is a single observation. If we $n$ observations with the same values $x$ for the features, but not all the same $y$ values, then the average loss for these $n$ samples is given by

$$ \frac{1}{n} \sum_i L_i =-\frac{1}{n} \sum_i \left[y_i \log f(x_i) + (1 - y_i) \log(1 - f(x_i)) \right] \\ = -\frac{k}{n} \log f(x) - \frac{n-k}{n} \log(1 - f(x)) \\ $$ where $k$ is the sum of $y$ for these $n$ samples. (We can collapse that sum because in this special case all the $x_i$ are identical.)

At this juncture, many people will leap to the conclusion that because the values $\frac{n-k}{n}$ and $\frac{k}{n}$ are probabilities (they're non-negative and they sum to 1 over disjoint events), then any time you have probabilities as $y$, it is appropriate to simply substitute the probabilities for the binary values $y$. This can bias the model. The demonstration is simple. Consider two distinct observations $x_1,x_2$ and their sums of binary labels $k_1=\sum_i y_1^{i},k_1=\sum_i y_2^{j}$ and sample counts $n_1,n_2$. Substituting the proportions is a blunder because this gives the loss as

$$\begin{align} \frac{1}{n_1}L_1 + \frac{1}{n_2}L_2 = &-\frac{k_1}{n_1} \log f(x_1) - \frac{n_1-k_1}{n_1} \log (1-f(x_1)) \\&- \frac{k_2}{n_2} \log f(x_2) - \frac{n_2-k_2}{n_2} \log (1-f(x_2)) \end{align}$$

It is a blunder because the proposed loss is not actually the average of all of the losses because the different sample sizes are neglected. This incorrect loss function yield produce a model that is biased in the sense that it weights all of the sample sizes equally, instead of giving more weight to the $x_i$ with larger $n_i$.

Correctly stated, the average loss is

$$\begin{align} \frac{1}{n_1+n_2} \sum_i L_i = &- \frac{k_1}{n_1+n_2} \log f(x_1) - \frac{n_1-k_1}{n_1+n_2} \log (1-f(x_1)) \\ &~- \frac{k_2}{n_1+n_2} \log f(x_2) - \frac{n_2-k_2}{n_1+n_2} \log (1-f(x_2)) \end{align}$$

The effect is to give more weight to the observations with larger sample sizes $n_i$. The blunder loss and the correct loss are only equivalent in the special case that all of the $n_i$ are equal; in this case, the blunder loss is a scalar multiple of the correct loss, which only changes the value of the loss function without changing the location of the minimum.

Answer 2

The answer given by @Sycorax is correct. However, it is worth mentioning that using (binary) cross-entropy in a regression task where the output values are in the range [0,1] is a valid and reasonable thing to do. Actually, it is used in image autoencoders (e.g. here and this paper). You might be interested to see a simple mathematical proof of why it works in this case in this answer.

Answer 3

Deep learning frameworks often mix models and losses and refer to the cross-entropy of a multinomial model with softmax nonlinearity by cross_entropy, which is misleading. In general, you can define cross-entropy for arbitrary models.

For a Gaussian model with varying mean but fixed diagonal covariance, it is equivalent to MSE. For a general covariance, cross-entropy would correspond to a squared Mahalanobis distance. For an exponential distribution, the cross-entropy loss would look like $$f_\theta(x) y - \log f_\theta(x),$$ where $y$ is continuous but non-negative. So yes, cross-entropy can be used for regression.

Answer 4

Unfortunately, the as of now accepted answer by @Sycorax, while detailed, is incorrect.

Actually, a prime example of regression through categorical cross-entropy -- Wavenet -- has been implemented in TensorFlow.

The principle is that you discretize your output space and then your model only predicts the respective bin; see Section 2.2 of the paper for an example in the sound modelling domain. So while technically the model performs classification, the eventual task solved is regression.

An obvious downside is, that you lose output resolution. However, this may not be a problem (at least I think that the Google's artificial assistant spoke a very humanly voice) or you can play around with some post-processing, e.g. interpolating between the most probable bin and it's two neighbours.

On the other hand, this approach makes the model much more powerful compared to the usual single-linear-unit output, i.e. allowing to express multi-modal predictions or to assess it's confidence. Note though that the latter can be naturally achieved by other means, e.g. by having an explicit (log)variance output as in Variational Autoencoders.

Anyway, this approach does not scale well to more-dimensional output, because then the size of the output layer grows exponentially, making it both computational and modelling issue..

Answer 5

I've revisited this question as I now disagree with the answer I previously accepted. Cross entropy loss CAN be used in regression (although it isn't common.)

It comes down to the fact that cross-entropy is a concept that only makes sense when comparing two probability distributions. You could consider a neural network which outputs a mean and standard deviation for a normal distribution as its prediction. It would then be punished more harshly for being more confident about bad predictions. So yes, it makes sense, but only if you're outputting a distribution in some sense. The link from @SiddharthShakya in a comment to my original question shows this.

Answer 6

Yes, sure.

What is Cross-Entropy?

Let's think about what is Cross-Entropy (CE). CE cost in the context of PyTorch or another Frameworks can mean a different thing compare to MATH. Originally cross-entropy is some form of KL-divergence between distributions: https://sites.google.com/site/burlachenkok/articles/properties-of-kl-divergence

Reasons:

1. In the context of Machine Learning, the very often first argument for CE is typically vector from probability simplex such that it has one component equal to one. So input for CE is probability mass function (p.m.f.) in a discrete case.

2. Inside PyTorch let's say there is an extra transformation called in STATS as "symmetric logistics transform" which $$\dfrac{exp(f_i(x)}{exp(f_1(x))+exp(f_2(x))+\dots + exp(f_k(x))}$$ What it's interesting it's a bijective mapping from Euclidane space into probabilistic simplex.

Now, what is a regression?

Regression in the context of probability theory means $E_{z}[y(x,z)|x]$ where by z I denote unobserved variables. In the context of machine learning, it means any predictor with an output single scalar variable or multiple scalar variables and it is not necessary conditional expectation.

---

So as you see - there is already too much confusion with various terminology and basic terms in STATS and ML.

If your model provides K scalar outputs (called sometimes logits) it can be plugged into CE with the symmetric logistic transformation. For the record - logits are just unbound scores from $\mathbb{R}$ such that class with maximum score is your prediction.

---

I think the answer you can do whatever you want, and people do crazy things with Loss in STATS, Optimization, and Deep Learning Applications.

I can not give you an exact answer because CE very often raised with classification models e.g. in Deep Learning or with Decision Trees. But if your question because you design such a system - it's better to allow more expressive power in Loss construction.