Skip to main content
4 of 10
edited body
dontloo
  • 16.8k
  • 9
  • 63
  • 88

loss function for data with noisy labels?

I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong).

The cross-entropy loss indeed works, but I was wondering are there any alternatives more effective in this case? or is cross-entropy loss the optimal?

I'm not sure but I'm thinking of somewhat "clipping" the cross-entropy loss, such that the loss for one data point will be no greater than some upper bound, will that work?

Thanks!

Update
According to Lucas's answer, I got the following for the derivatives for the prediction output $y$ and input of the softmax function $z$. So I guess essentially it is giving an upper bound and lower bound to the refined output $p$ (and therefore the loss as well). Please let me know if I'm wrong. Thanks!
$$p_i=0.3/N+0.7y_i$$ $$l=-\sum t_i\log(p_i)$$ $$\frac{\partial l}{\partial y_i}=-t_i\frac{\partial\log(p_i)}{\partial p_i}\frac{\partial p_i}{\partial y_i}=-0.7\frac{t_i}{p_i}$$ $$\frac{\partial l}{\partial z_i}=0.7\sum_j\frac{t_j}{p_j}\frac{\partial y_j}{\partial z_i}=0.7(y_i\sum_jt_j\frac{y_j}{p_j}-t_i\frac{y_i}{p_i})$$

dontloo
  • 16.8k
  • 9
  • 63
  • 88