I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong).

The cross-entropy loss indeed works, but I was wondering are there any alternatives more effective in this case? or is cross-entropy loss the optimal?

I'm not sure but I'm thinking of somewhat "clipping" the cross-entropy loss, such that the loss for one data point will be no greater than some upper bound, will that work?

Thanks!

Update
According to Lucas' answer, I got the following for the derivatives for the prediction output $y$ and input of the softmax function $z$. So I guess essentially it is adding a smoothing term $\frac{3}{7N}$ to the derivatives.
$$p_i=0.3/N+0.7y_i$$ $$l=-\sum t_i\log(p_i)$$ $$\frac{\partial l}{\partial y_i}=-t_i\frac{\partial\log(p_i)}{\partial p_i}\frac{\partial p_i}{\partial y_i}=-0.7\frac{t_i}{p_i}=-\frac{t_i}{\frac{3}{7N}+y_i}$$ $$\frac{\partial l}{\partial z_i}=0.7\sum_j\frac{t_j}{p_j}\frac{\partial y_j}{\partial z_i}=y_i\sum_jt_j\frac{y_j}{\frac{3}{7N}+y_j}-t_i\frac{y_i}{\frac{3}{7N}+y_i}$$ Derivatives for the original cross-entropy loss: $$\frac{\partial l}{\partial y_i}=-\frac{t_i}{y_i}$$ $$\frac{\partial l}{\partial z_i}=y_i-t_i$$ Please let me know if I'm wrong. Thanks!

Update
I happened to find a paper by Google that applies the same formula as in Lucas' answer but with different interpretations. Rethinking the Inception Architecture for Computer Vision Section 7 Model Regularization via Label Smoothing

This (the cross entropy loss), however, can cause two problems. First, it may result in over-fitting: if the model learns to assign full probability to the groundtruth label for each training example, it is not guaranteed to generalize. Second, it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $∂l/∂z_k$, reduces the ability of the model to adapt. Intuitively, this happens because the model becomes too confident about its predictions.

But instead of adding the smoothing term to the predictions, they added it to the ground truth, which turned out to be helpful.

In our ImageNet experiments with K = 1000 classes, we used u(k) = 1/1000 and $\epsilon$ = 0.1. For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error.

I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong).

The cross-entropy loss indeed works, but I was wondering are there any alternatives more effective in this case? or is cross-entropy loss the optimal?

I'm not sure but I'm thinking of somewhat "clipping" the cross-entropy loss, such that the loss for one data point will be no greater than some upper bound, will that work?

Thanks!

Update
According to Lucas' answer, I got the following for the derivatives for the prediction output $y$ and input of the softmax function $z$. So I guess essentially it is adding a smoothing term $\frac{3}{7N}$ to the derivatives.
$$p_i=0.3/N+0.7y_i$$ $$l=-\sum t_i\log(p_i)$$ $$\frac{\partial l}{\partial y_i}=-t_i\frac{\partial\log(p_i)}{\partial p_i}\frac{\partial p_i}{\partial y_i}=-0.7\frac{t_i}{p_i}=-\frac{t_i}{\frac{3}{7N}+y_i}$$ $$\frac{\partial l}{\partial z_i}=0.7\sum_j\frac{t_j}{p_j}\frac{\partial y_j}{\partial z_i}=y_i\sum_jt_j\frac{y_j}{\frac{3}{7N}+y_j}-t_i\frac{y_i}{\frac{3}{7N}+y_i}$$ Derivatives for the original cross-entropy loss: $$\frac{\partial l}{\partial y_i}=-\frac{t_i}{y_i}$$ $$\frac{\partial l}{\partial z_i}=y_i-t_i$$ Please let me know if I'm wrong. Thanks!

Update
I just happened to read a paper by Google that applies the same formula as in Lucas' answer but with different interpretations.

In Section 7 Model Regularization via Label Smoothing

This (the cross entropy loss), however, can cause two problems. First, it may result in over-fitting: if the model learns to assign full probability to the groundtruth label for each training example, it is not guaranteed to generalize. Second, it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $∂l/∂z_k$, reduces the ability of the model to adapt. Intuitively, this happens because the model becomes too confident about its predictions.

But instead of adding the smoothing term to the predictions, they added it to the ground truth, which turned out to be helpful.

In our ImageNet experiments with K = 1000 classes, we used u(k) = 1/1000 and $\epsilon$ = 0.1. For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error.

I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong).

The cross-entropy loss indeed works, but I was wondering are there any alternatives more effective in this case? or is cross-entropy loss the optimal?

I'm not sure but I'm thinking of somewhat "clipping" the cross-entropy loss, such that the loss for one data point will be no greater than some upper bound, will that work?

Thanks!

Update
According to Lucas' answer, I got the following for the derivatives for the prediction output $y$ and input of the softmax function $z$. So I guess essentially it is adding a smoothing term $\frac{3}{7N}$ to the derivatives.
$$p_i=0.3/N+0.7y_i$$ $$l=-\sum t_i\log(p_i)$$ $$\frac{\partial l}{\partial y_i}=-t_i\frac{\partial\log(p_i)}{\partial p_i}\frac{\partial p_i}{\partial y_i}=-0.7\frac{t_i}{p_i}=-\frac{t_i}{\frac{3}{7N}+y_i}$$ $$\frac{\partial l}{\partial z_i}=0.7\sum_j\frac{t_j}{p_j}\frac{\partial y_j}{\partial z_i}=y_i\sum_jt_j\frac{y_j}{\frac{3}{7N}+y_j}-t_i\frac{y_i}{\frac{3}{7N}+y_i}$$ Derivatives for the original cross-entropy loss: $$\frac{\partial l}{\partial y_i}=-\frac{t_i}{y_i}$$ $$\frac{\partial l}{\partial z_i}=y_i-t_i$$ Please let me know if I'm wrong. Thanks!

Update
I happened to find a paper by Google that applies the same formula as in Lucas' answer but with different interpretations. Rethinking the Inception Architecture for Computer Vision Section 7 Model Regularization via Label Smoothing

This (the cross entropy loss), however, can cause two problems. First, it may result in over-fitting: if the model learns to assign full probability to the groundtruth label for each training example, it is not guaranteed to generalize. Second, it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $∂l/∂z_k$, reduces the ability of the model to adapt. Intuitively, this happens because the model becomes too confident about its predictions.

But instead of adding the smoothing term to the predictions, they added it to the ground truth and it turned out to be helpful.

In our ImageNet experiments with K = 1000 classes, we used u(k) = 1/1000 and $\epsilon$ = 0.1. For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error.

I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong).

The cross-entropy loss indeed works, but I was wondering are there any alternatives more effective in this case? or is cross-entropy loss the optimal?

I'm not sure but I'm thinking of somewhat "clipping" the cross-entropy loss, such that the loss for one data point will be no greater than some upper bound, will that work?

Thanks!

Update
According to Lucas' answer, I got the following for the derivatives for the prediction output $y$ and input of the softmax function $z$. So I guess essentially it is adding a smoothing term $\frac{3}{7N}$ to the derivatives.
$$p_i=0.3/N+0.7y_i$$ $$l=-\sum t_i\log(p_i)$$ $$\frac{\partial l}{\partial y_i}=-t_i\frac{\partial\log(p_i)}{\partial p_i}\frac{\partial p_i}{\partial y_i}=-0.7\frac{t_i}{p_i}=-\frac{t_i}{\frac{3}{7N}+y_i}$$ $$\frac{\partial l}{\partial z_i}=0.7\sum_j\frac{t_j}{p_j}\frac{\partial y_j}{\partial z_i}=y_i\sum_jt_j\frac{y_j}{\frac{3}{7N}+y_j}-t_i\frac{y_i}{\frac{3}{7N}+y_i}$$ Derivatives for the original cross-entropy loss: $$\frac{\partial l}{\partial y_i}=-\frac{t_i}{y_i}$$ $$\frac{\partial l}{\partial z_i}=y_i-t_i$$ Please let me know if I'm wrong. Thanks!

Update
I happened to find a paper by Google that applies the same formula as in Lucas' answer but with different interpretations. Rethinking the Inception Architecture for Computer Vision Section 7 Model Regularization via Label Smoothing

This (the cross entropy loss), however, can cause two problems. First, it may result in over-fitting: if the model learns to assign full probability to the groundtruth label for each training example, it is not guaranteed to generalize. Second, it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $∂l/∂z_k$, reduces the ability of the model to adapt. Intuitively, this happens because the model becomes too confident about its predictions.

But instead of adding the smoothing term to the predictions, they added it to the ground truth, which turned out to be helpful.

In our ImageNet experiments with K = 1000 classes, we used u(k) = 1/1000 and $\epsilon$ = 0.1. For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error.

I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong).

The cross-entropy loss indeed works, but I was wondering are there any alternatives more effective in this case? or is cross-entropy loss the optimal?

I'm not sure but I'm thinking of somewhat "clipping" the cross-entropy loss, such that the loss for one data point will be no greater than some upper bound, will that work?

Thanks!

Update
According to Lucas' answer, I got the following for the derivatives for the prediction output $y$ and input of the softmax function $z$. So I guess essentially it is adding a smoothing term $\frac{3}{7N}$ to the derivatives.
$$p_i=0.3/N+0.7y_i$$ $$l=-\sum t_i\log(p_i)$$ $$\frac{\partial l}{\partial y_i}=-t_i\frac{\partial\log(p_i)}{\partial p_i}\frac{\partial p_i}{\partial y_i}=-0.7\frac{t_i}{p_i}=-\frac{t_i}{\frac{3}{7N}+y_i}$$ $$\frac{\partial l}{\partial z_i}=0.7\sum_j\frac{t_j}{p_j}\frac{\partial y_j}{\partial z_i}=y_i\sum_jt_j\frac{y_j}{\frac{3}{7N}+y_j}-t_i\frac{y_i}{\frac{3}{7N}+y_i}$$ Derivatives for the original cross-entropy loss: $$\frac{\partial l}{\partial y_i}=-\frac{t_i}{y_i}$$ $$\frac{\partial l}{\partial z_i}=y_i-t_i$$ Please let me know if I'm wrong. Thanks!

Update
I happened to find a paper by Google that applies the same formula as in Lucas' answer but with different interpretations. Rethinking the Inception Architecture for Computer Vision Section 7 Model Regularization via Label Smoothing

This (the cross entropy loss), however, can cause two problems. First, it may result in over-fitting: if the model learns to assign full probability to the groundtruth label for each training example, it is not guaranteed to generalize. Second, it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $∂l/∂z_k$, reduces the ability of the model to adapt. Intuitively, this happens because the model becomes too confident about its predictions.

It turned out to be helpful.

In our ImageNet experiments with K = 1000 classes, we used u(k) = 1/1000 and $\epsilon$ = 0.1. For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error.

I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong).

The cross-entropy loss indeed works, but I was wondering are there any alternatives more effective in this case? or is cross-entropy loss the optimal?

I'm not sure but I'm thinking of somewhat "clipping" the cross-entropy loss, such that the loss for one data point will be no greater than some upper bound, will that work?

Thanks!

Update
According to Lucas' answer, I got the following for the derivatives for the prediction output $y$ and input of the softmax function $z$. So I guess essentially it is adding a smoothing term $\frac{3}{7N}$ to the derivatives.
$$p_i=0.3/N+0.7y_i$$ $$l=-\sum t_i\log(p_i)$$ $$\frac{\partial l}{\partial y_i}=-t_i\frac{\partial\log(p_i)}{\partial p_i}\frac{\partial p_i}{\partial y_i}=-0.7\frac{t_i}{p_i}=-\frac{t_i}{\frac{3}{7N}+y_i}$$ $$\frac{\partial l}{\partial z_i}=0.7\sum_j\frac{t_j}{p_j}\frac{\partial y_j}{\partial z_i}=y_i\sum_jt_j\frac{y_j}{\frac{3}{7N}+y_j}-t_i\frac{y_i}{\frac{3}{7N}+y_i}$$ Derivatives for the original cross-entropy loss: $$\frac{\partial l}{\partial y_i}=-\frac{t_i}{y_i}$$ $$\frac{\partial l}{\partial z_i}=y_i-t_i$$ Please let me know if I'm wrong. Thanks!

Update
I happened to find a paper by Google that applies the same formula as in Lucas' answer but with different interpretations. Rethinking the Inception Architecture for Computer Vision Section 7 Model Regularization via Label Smoothing

This (the cross entropy loss), however, can cause two problems. First, it may result in over-fitting: if the model learns to assign full probability to the groundtruth label for each training example, it is not guaranteed to generalize. Second, it encourages the differences between the largest logit and all others to become large, and this, combined with the bounded gradient $∂l/∂z_k$, reduces the ability of the model to adapt. Intuitively, this happens because the model becomes too confident about its predictions.

But instead of adding the smoothing term to the predictions, they added it to the ground truth and it turned out to be helpful.

In our ImageNet experiments with K = 1000 classes, we used u(k) = 1/1000 and $\epsilon$ = 0.1. For ILSVRC 2012, we have found a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error.