Dealing with label noise (Regression, NLP)

Question

For my school project, my group is tackling this Kaggle challenge (assign reading level based on passage).

commonlitreadabilityprize

However, it seems there is some label noise (examples below, lower score means more difficult)

https://i.sstatic.net/tNwKU.jpg

https://i.sstatic.net/vEyuO.jpg

https://i.sstatic.net/w9KcE.jpg

What are some good ways with identifying and/or correcting mislabeled regression scores for textual data?

Since this is for a Deep Learning course, deep methods would be preferred.

Answer 1

There are a lot more prior works on noisy labels than noisy regression values; we could adapt some ideas from them.

Some of the easier categories of techniques (and some ways to use them) are:

1. Robust architecture
   • We take the regression value and corrupt it (e.g. sample from Gaussian distribution with the given regression value and the standard deviation). Then on top of the base model you already have, we add an additional layer that takes in the input from the base model and predicts the noisy value. The loss function will be the difference between the generated noisy value and the predictions. When evaluating, we will use the prediction of the base model.
2. Robust regularization
   • We can conduct data augmentation by augmenting our training data to harder and easier versions and include them in our training data. This is usually based on heuristics which would require you to have some knowledge in linguistic (or if you are lucky, there might be prior works that you can use easily).
   • A similar idea would be to source for additional datasets. e.g. if you believe that classic literature works (e.g. Shakespeares) use harder-to-read sentence structures and vocabularies, you can include them in your datasets as hard examples. Conversely, you could do the same by including children's books as easy examples. By doing so, you are hoping to bais the NN to capture different styles of sentence structures and vocabularies and use them in the prediction.
3. Loss adjustment
   • You can try to reweight the losses by the standard deviation of the regression to account for the uncertainty around the target. More specifically, data points with higher standard deviation should have smaller loss values (e.g. multiply by 1/std.dev - do play around with different weighting functions).
4. Sample selection
   • You can try to do co-teaching. Train 2 NN simultaneously, at each mini-batch, for each network, sample the data points with a smaller loss. You will end up with 2 sets of data containing the smaller loss instances. You will calculate the gradient for a model using the data sampled from the other model.