Is categorical cross entropy loss wasting training data?

Question

Given this categorical cross entropy loss function $-\text{sum}(y * \text{log}(\hat{y}))$, where $y$ is a one-hot word vector and $\hat{y}$ is a vector of class probabilities, this loss function will ignore the model predictions where $y=0$.

For example, let $y = [0, 1, 0]$, and $\hat{y} = [.7, .6, .2]$

Then

$$ -\text{sum}(y * \text{log}(\hat{y})) = -1 * 1 * \text{log}(.6) = .2 $$

This ignores the $.7$ in $\hat{y}$ which is a large probability and is wrong right? Is this a drawback of using categorical cross entropy loss?

Would it be better to modify categorical cross-entropy loss to have two terms:

$$ -1 * \text{sum} ( y * \text{log}(\hat{y})) + -1 * \alpha * \text{sum} (\text{Inv}(y) * \text{log}(\hat{y})) $$

Here $\text{Inv}(y)$ would be $1$ when $y$ is $0$ and $0$ when $y$ is $1$.
$\alpha$ would be a constant to weight the loss term to allow for incorrect predictions.

Would this not penalize incorrect predictions from all classes? In the original categorical cross entropy loss are we not wasting that information?

Answer 1

The gold standard optimization criterion is the log likelihood (including penalized log likelihood) and the Bayesian counterpart log posterior (log likelihood + log prior). In the binary outcome $Y$ case an observation observing $Y=y$ when the current predicted probability is $p$ has a log-likelihood of $y \log(p) + (1 - y) \log(1 - p)$. Observations contribute to either the first or the second part to the log-likelihood. No observation is ignored. For multi-category $Y$ (multinomial) results are similar. The probability element for $Y=y, y=1, 2, \ldots, k$ is $p_{y} = \Pr(Y=y)$.