114

KL divergence is a natural way to measure the difference between two probability distributions. The entropy $H(p)$ of a distribution $p$ gives the minimum possible number of bits per message that would be needed (on average) to losslessly encode events drawn from $p$. Achieving this bound would require using an optimal code designed for $p$, which assigns ...


75

Bernoulli$^*$ cross-entropy loss is a special case of categorical cross-entropy loss for $m=2$. $$ \begin{align} \mathcal{L}(\theta) &= -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^m y_{ij}\log(p_{ij}) \\ &= -\frac{1}{n}\sum_{i=1}^n \left[y_i \log(p_i) + (1-y_i) \log(1-p_i)\right] \end{align} $$ Where $i$ indexes samples/observations and $j$ indexes classes, ...


69

You will need some conditions to claim the equivalence between minimizing cross entropy and minimizing KL divergence. I will put your question under the context of classification problems using cross entropy as loss functions. Let us first recall that entropy is used to measure the uncertainty of a system, which is defined as \begin{equation} S(v)=-\sum_ip(...


60

Both, categorical cross entropy and sparse categorical cross entropy have the same loss function which you have mentioned above. The only difference is the format in which you mention $Y_i$ (i,e true labels). If your $Y_i$'s are one-hot encoded, use categorical_crossentropy. Examples (for a 3-class classification): [1,0,0] , [0,1,0], [0,0,1] But if your $...


47

If you are using keras, just put sigmoids on your output layer and binary_crossentropy on your cost function. If you are using tensorflow, then can use sigmoid_cross_entropy_with_logits. But for my case this direct loss function was not converging. So I ended up using explicit sigmoid cross entropy loss $(y \cdot \ln(\text{sigmoid}(\text{logits})) + (1-y) \...


46

Note: I am not an expert on backprop, but now having read a bit, I think the following caveat is appropriate. When reading papers or books on neural nets, it is not uncommon for derivatives to be written using a mix of the standard summation/index notation, matrix notation, and multi-index notation (include a hybrid of the last two for tensor-tensor ...


45

One compelling reason for using cross-entropy over dice-coefficient or the similar IoU metric is that the gradients are nicer. The gradients of cross-entropy wrt the logits is something like $p - t$, where $p$ is the softmax outputs and $t$ is the target. Meanwhile, if we try to write the dice coefficient in a differentiable form: $\frac{2pt}{p^2+t^2}$ or $\...


39

There are three kinds of classification tasks: Binary classification: two exclusive classes Multi-class classification: more than two exclusive classes Multi-label classification: just non-exclusive classes Here, we can say In the case of (1), you need to use binary cross entropy. In the case of (2), you need to use categorical cross entropy. In ...


39

Let the data be $\mathbf{x}=(x_1, \ldots, x_n)$. Write $F(\mathbf{x})$ for the empirical distribution. By definition, for any function $f$, $$\mathbb{E}_{F(\mathbf{x})}[f(X)] = \frac{1}{n}\sum_{i=1}^n f(x_i).$$ Let the model $M$ have density $e^{f(x)}$ where $f$ is defined on the support of the model. The cross-entropy of $F(\mathbf{x})$ and $M$ is ...


27

As summarized by @shimao and @cherub, one cannot say apriori which one will work better on a particular dataset. The correct way is to try both and compare the results. Also, note that when it comes to segmentation, it is not so easy to "compare the results": IoU based measures like dice coefficient cover only some aspects of the quality of the segmentation; ...


26

These three definitions are essentially the same. 1) The Tensorflow introduction, $$C = -\frac{1}{n} \sum\limits_x\sum\limits_{j} (y_j \ln a_j).$$ 2) For binary classifications $j=2$, it becomes $$C = -\frac{1}{n} \sum\limits_x (y_1 \ln a_1 + y_2 \ln a_2)$$ and because of the constraints $\sum_ja_j=1$ and $\sum_jy_j=1$, it can be rewritten as $$C = -\frac{...


26

While @GeoMatt22's answer is correct, I personally found it very useful to reduce the problem to a toy example and draw a picture: I then defined the operations each node was computing, treating the $h$'s and $w$'s as inputs to a "network" ($\mathbf{t}$ is a one-hot vector representing the class label of the data point): $$L=-t_1\log o_1 -t_2\log o_2$$ $$...


25

I think the best answer to this is that the cross-entropy loss function is just not well-suited to this particular task. In taking this approach, you are essentially saying the true MNIST data is binary, and your pixel intensities represent the probability that each pixel is 'on.' But we know this is not actually the case. The incorrectness of this implicit ...


22

Binary cross-entropy is for multi-label classifications, whereas categorical cross entropy is for multi-class classification where each example belongs to a single class.


22

log base e and log base 2 are only a constant factor off from each other: $$\frac{\log_e{n}}{\log_2{n}} = \frac{\log_e{2}}{\log_e e} = \log_e 2$$ Therefore using one over the other scales the entropy by a constant factor. When using log base 2, the unit of entropy is bits, where as with natural log, the unit is nats. One isn't better than the other. It's ...


22

I suppose it is because the models usually work with the samples packed in mini-batches. For KL divergence and Cross-Entropy, their relation can be written as $$H(q, p) = D_{KL}(p, q)+H(p) = -\sum_i{p_ilog(q_i)}$$ so have $$D_{KL}(p, q) = H(q, p) - H(p)$$ From the equation, we could see that KL divergence can depart into a Cross-Entropy of p and q (the first ...


16

Here's a worked example in the case of iid binary data, each with a success/failure recorded as $y_i \in \{0,1\}$. For labels $y_i\in \{0,1\}$, the likelihood of some binary data under the Bernoulli model with parameters $\theta$ is $$ \mathcal{L}(\theta) = \prod_{i=1}^n p(y_i=1|\theta)^{y_i}p(y_i=0|\theta)^{1-y_i}\\ $$ whereas the log-likelihood is $$ \log\...


15

UPDATE (18/04/18): The old answer still proved to be useful on my model. The trick is to model the partition function and the distribution separately, thus exploiting the power of softmax. Consider your observation vector $y$ to contain $m$ labels. $y_{im}=\delta_{im}$ (1 if sample i contains label m, 0 otherwise). So the objective would be to to model ...


14

No, it doesn't make sense to use TensorFlow functions like tf.nn.sigmoid_cross_entropy_with_logits for a regression task. In TensorFlow, “cross-entropy” is shorthand (or jargon) for “categorical cross entropy.” Categorical cross entropy is an operation on probabilities. A regression problem attempts to predict continuous outcomes, rather than classifications....


13

Suppose that we are trying to infer the parametric distribution $p(y|\Theta(X))$, where $\Theta(X)$ is a vector output inverse link function with $[\theta_1,\theta_2,...,\theta_M]$. We have a neural network at hand with some topology we decided. The number of outputs at the output layer matches the number of parameters we would like to infer (it may be less ...


12

For binary classification one way to encode the probability of an output is $p^y(1-p)^{1-y}$, if y is encoded as 0 or 1. This is the likelihood function and it’s meaning is with probability p we output 0 and with probability 1-p if output is 1. Now you have a sample and you want to find p which best fits your data. One way is to find the maximum likelihood ...


12

The earliest I have been able to find is Good, I. J. “Rational Decisions.” Journal of the Royal Statistical Society. Series B (Methodological), vol. 14, no. 1, 1952, pp. 107–114. JSTOR, www.jstor.org/stable/2984087 Look at section 8, "Fair Fees": By itself $\log p_1$ (or $\log(1 - p_1)$) is a measure of the merit of a probability estimate I ...


11

Cross entropy is defined on probability distributions, not single values. The reason it works for classification is that classifier output is (often) a probability distribution over class labels. For example, the outputs of logistic/softmax functions are interpreted as probabilities. The observed class label is also treated as a probability distribution: the ...


11

Andrew Ng explains the intuition behind using cross-entropy as a cost function in his ML Coursera course under the logistic regression module, specifically at this point in time with the mathematical expression: $$\text{Cost}\left(h_\theta(x),y\right)=\left\{ \begin{array}{l} -\log\left(h_\theta(x)\right) \quad \quad\quad \text{if $y =1$}\\ -\log\left(1 -h_\...


11

The internet has told me that when using Softmax combined with cross entropy, Step 1 simply becomes $\frac{\partial E} {\partial z_j} = o_j - t_j$ where $t$ is a one-hot encoded target output vector. Is this correct? Yes. Before going through the proof, let me change the notation to avoid careless mistakes in translation: Notation: I'll follow the ...


10

The answer given by @Sycorax is correct. However, it is worth mentioning that using (binary) cross-entropy in a regression task where the output values are in the range [0,1] is a valid and reasonable thing to do. Actually, it is used in image autoencoders (e.g. here and this paper). You might be interested to see a simple mathematical proof of why it works ...


10

A mistake in your code: $$-\frac{1}{N}\sum_{i=1}^N [\color{red}{\hat{y}_i} \log(\hat{y}_i)+(1-y_i) \log(1-\hat{y}_i)]$$ It should be $$-\frac{1}{N}\sum_{i=1}^N [\color{blue}{y_i} \log(\hat{y}_i)+(1-y_i) \log(1-\hat{y}_i)]$$ Your code: result.append([y_pred[i][j] * math.log(y_pred[i][j]) + (1 - y_true[i][j]) * math.log(1 - y_pred[i][j]) for j in range(...


10

If we view minimizing cross entropy as equivalent to maximizing the log-likelihood of the same model, then I believe we can go as far back as RA Fisher. This places the date between 1912 and 1922, depending on how well-developed you wish the theory to be; see discussion in John Aldrich "R. A. Fisher and the Making of Maximum Likelihood 1912 – 1922" ...


9

For readers of the Deep Learning book, I would like to add to the excellent accepted answer that the authors explain their statement in detail in section 5.5.1 namely the Example: Linear Regression as Maximum Likelihood. There, they list exactly the constraint mentioned in the accepted answer: $p(y | x) = \mathcal{N}\big(y; \hat{y}(x; w), \sigma^2\big)$....


9

The Wikipedia page has left out a few steps. When they say "the likelihood of the training set", they just mean "the probability of the training set given some parameter values": $$ L(\theta | x_1, ..., x_n) = p(x_1, ..., x_n | \theta) $$ The data $x_k$ in the training set is assumed conditionally independent, so that $$ p(x_1, ..., x_n | \theta) = \...


Only top voted, non community-wiki answers of a minimum length are eligible