2
$\begingroup$

Suppose we have two greyscale images which are flattened to 1d arrays: $y=(y_1, y_2, \ldots, y_n)$ and $\hat{y} = (\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_n)$ with pixel values in $[0,1]$. How exactly do we use cross-entropy to compare these images?

The definition of cross entropy leads me to believe that we should compute $$-\sum_{i} y_i \log \hat{y}_i,$$ but in the machine learning context I usually see loss functions using "binary" cross entropy, which I believe is $$ -\sum_i y_i \log \hat{y}_i - \sum_i (1-y_i) \log (1-\hat{y}_i).$$

Can someone please clarify this for me?

$\endgroup$
  • $\begingroup$ The source of the confusion is that $i$ in the first expression indexes over classes while $i$ in the second expression indexes over pixels. $\endgroup$ – shimao May 31 '18 at 3:25
  • $\begingroup$ @shimao What are the classes in this case? I thought the "binary" in binary-cross-entropy referred to having two classes? $\endgroup$ – theQman May 31 '18 at 12:57
2
$\begingroup$

The cross-entropy between a single label and prediction would be

$$L = -\sum_{c \in C} y_{c} \log \hat y_{c}$$

where $C$ is the set of all classes. This is the first expression in your post. However, we need to sum over all pixels in an image to apply this:

$$L = -\sum_{i \in I} \sum_{c \in C} y_{i,c} \log \hat y_{i,c}$$

where $I$ is the set of pixels in an image and with $y_{i,c}$ being an indicator variable for whether the $i$th pixel is in class $c$.

In the binary case, we only have two classes: $0$ and $1$.

$$L = -\sum_{i \in I} \left( y_{i,0} \log \hat y_{i,0} + y_{i,1} \log \hat y_{i,1} \right)$$

Since $y_{i,0} + y_{i,1}$ must necessarily sum to 1, we can also just drop the class indices and denote $y_i = y_{i,0}$ and $1-y_i = y_{i,1}$.

$$L = -\sum_{i \in I} \left( y_i \log \hat{y_i} + (1-y_i) \log (1-\hat y_i) \right)$$

This is where the second equation in your post comes from.

$\endgroup$
  • $\begingroup$ So is there an implicit assumption along the lines of "black means class 0, white means class 1" and then some grey pixel value of 0.1 means the pixel very likely belongs to the black class? I think what is throwing me off is that usually we have $y_c$ or $y_{i,c}$ being 0 or 1, but here both $y_c$ and $\hat{y}_c$ can be real numbers. $\endgroup$ – theQman May 31 '18 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.