Skip to main content
2 of 2
typo
Lucas
  • 6.2k
  • 34
  • 39

The right thing to do here is to change the model, not the loss. Your goal is still to correctly classify as many data points as possible (which determines the loss), but your assumptions about the data have changed (which are encoded in a statistical model, the neural network in this case).

Let $\mathbf{p}_t$ be a vector of class probabilities produced by the neural network and $\ell(y_t, \mathbf{p}_t)$ be the cross-entropy loss for label $y_t$. To explicitly take into account the assumption that 30% of the labels are noise (assumed to be uniformly random), we could change our model to produce

$$\mathbf{\tilde p}_t = 0.3/N + 0.7 \mathbf{p}_t$$

instead and optimize

$$\sum_t \ell(y_t, 0.3/N + 0.7 \mathbf{p}_t),$$

where $N$ is the number of classes. This will actually behave somewhat according to your intuition, limiting the loss to be finite.

Lucas
  • 6.2k
  • 34
  • 39