# Is categorical cross entropy loss wasting training data?

Given this categorical cross entropy loss function $$-\text{sum}(y * \text{log}(\hat{y}))$$, where $$y$$ is a one-hot word vector and $$\hat{y}$$ is a vector of class probabilities, this loss function will ignore the model predictions where $$y=0$$.

For example, let $$y = [0, 1, 0]$$, and $$\hat{y} = [.7, .6, .2]$$

Then

$$-\text{sum}(y * \text{log}(\hat{y})) = -1 * 1 * \text{log}(.6) = .2$$

This ignores the $$.7$$ in $$\hat{y}$$ which is a large probability and is wrong right? Is this a drawback of using categorical cross entropy loss?

Would it be better to modify categorical cross-entropy loss to have two terms:

$$-1 * \text{sum} ( y * \text{log}(\hat{y})) + -1 * \alpha * \text{sum} (\text{Inv}(y) * \text{log}(\hat{y}))$$

Here $$\text{Inv}(y)$$ would be $$1$$ when $$y$$ is $$0$$ and $$0$$ when $$y$$ is $$1$$.
$$\alpha$$ would be a constant to weight the loss term to allow for incorrect predictions.

Would this not penalize incorrect predictions from all classes? In the original categorical cross entropy loss are we not wasting that information?

• I don't quite understand your example. Do you have a single instance and three possible classes, out of which the true observation is the second one (but your probabilistic predictions do not sum to 1)? Or do you have three instances, and your $\hat{y}$ is the predicted probability that the $i$-th one is equal to $1$? Please clarify. Commented Aug 3, 2023 at 11:42
• Your intuition is correct, but you're using the wrong loss function. If you do the algebra, you can show the correct one to be $\log{\hat p(y_\text{true})}$, then 'never wasting training data' Commented Aug 3, 2023 at 11:50

The gold standard optimization criterion is the log likelihood (including penalized log likelihood) and the Bayesian counterpart log posterior (log likelihood + log prior). In the binary outcome $$Y$$ case an observation observing $$Y=y$$ when the current predicted probability is $$p$$ has a log-likelihood of $$y \log(p) + (1 - y) \log(1 - p)$$. Observations contribute to either the first or the second part to the log-likelihood. No observation is ignored. For multi-category $$Y$$ (multinomial) results are similar. The probability element for $$Y=y, y=1, 2, \ldots, k$$ is $$p_{y} = \Pr(Y=y)$$.