I recognize that this is a stats forum and the expectation is a focus on mathematical derivations, but if it can be helpful and you're using Python, there is a package for classification with noisy labels called cleanlab: https://github.com/cgnorthcutt/cleanlab/. (Disclaimer: I am the author of cleanlab)

The cleanlab Python package, pip install cleanlab, for which I am an author, finds label errors in datasets and supports classification/learning with noisy labels. It works with scikit-learn, PyTorch, Tensorflow, FastText, etc.

For learning with noisy labels.

# Code taken from https://github.com/cgnorthcutt/cleanlab
from cleanlab.classification import LearningWithNoisyLabels
from sklearn.linear_model import LogisticRegression

# Learning with noisy labels in 3 lines of code.

# Wrap around any classifier. Works with sklearn/pyTorch/Tensorflow/FastText/etc.
lnl = LearningWithNoisyLabels(clf=LogisticRegression())
lnl.fit(X = X_train_data, s = train_noisy_labels)
# Estimate the predictions you would have gotten by training with *no* label errors.
predicted_test_labels = lnl.predict(X_test)

To find label errors in your dataset.

from cleanlab.latent_estimation import estimate_cv_predicted_probabilities

# Find the indices of label errors in 2 lines of code.

probabilities = estimate_cv_predicted_probabilities(
    X_train_data, 
    train_noisy_labels, 
    clf=LogisticRegression(),
)
label_error_indices = get_noise_indices(
    s = train_noisy_labels, 
    psx = probabilities, 
)

Some examples with FastText (NLP) and PyTorch (MNIST AlexNet).

Documentation: https://l7.curtisnorthcutt.com/cleanlab-python-package

This approach has been proven to work in many realistic settings with substantive theory for models with error in every predicted probability output for every example.

The approach counts up an unnormalized estimate of the joint distribution of true labels and noisy/given labels. Using that estimate, it finds the label errors in the dataset so you can train on clean data. It has been shown to compare favorably to most methods and works for any dataset you can train a classifier on and for most data formats, ML and deep learning frameworks, and data modalities, e.g. image, text, tabular, and audio data. I am an author on this package.

• link to python package: https://github.com/cleanlab/cleanlab

Find label issues in 1 line of code

from cleanlab.classification import CleanLearning
from cleanlab.filter import find_label_issues

# Option 1 - works with sklearn-compatible models - just input the data and labels ツ
label_issues_info = CleanLearning(clf=sklearn_compatible_model).find_label_issues(data, labels)

# Option 2 - works with ANY ML model - just input the model's predicted probabilities
ordered_label_issues = find_label_issues(
    labels=labels,
    pred_probs=pred_probs,  # out-of-sample predicted probabilities from any model
    return_indices_ranked_by='self_confidence',
)

Train a model as if the dataset did not have errors -- 3 lines of code

from sklearn.linear_model import LogisticRegression
from cleanlab.classification import CleanLearning

cl = CleanLearning(clf=LogisticRegression())  # any sklearn-compatible classifier
cl.fit(train_data, labels)

# Estimate the predictions you would have gotten if you trained without mislabeled data.
predictions = cl.predict(test_data)

• Journal of AI Research (with theory to prove it works): https://arxiv.org/abs/1911.00068publication
• errors found using cleanlab: https://labelerrors.com/
• Documentation and runnable tutorials for cleanlab: https://docs.cleanlab.ai/

I recognize that this is a stats forum and the expectation is a focus on mathematical derivations, but if it can be helpful and you're using Python, there is a package for classification with noisy labels called cleanlab: https://github.com/cgnorthcutt/cleanlab/.

The cleanlab Python package, pip install cleanlab, for which I am an author, finds label errors in datasets and supports classification/learning with noisy labels. It works with scikit-learn, PyTorch, Tensorflow, FastText, etc.

For learning with noisy labels.

# Code taken from https://github.com/cgnorthcutt/cleanlab
from cleanlab.classification import LearningWithNoisyLabels
from sklearn.linear_model import LogisticRegression

# Learning with noisy labels in 3 lines of code.

# Wrap around any classifier. Works with sklearn/pyTorch/Tensorflow/FastText/etc.
lnl = LearningWithNoisyLabels(clf=LogisticRegression())
lnl.fit(X = X_train_data, s = train_noisy_labels)
# Estimate the predictions you would have gotten by training with *no* label errors.
predicted_test_labels = lnl.predict(X_test)

To find label errors in your dataset.

from cleanlab.latent_estimation import estimate_cv_predicted_probabilities

# Find the indices of label errors in 2 lines of code.

probabilities = estimate_cv_predicted_probabilities(
    X_train_data, 
    train_noisy_labels, 
    clf=LogisticRegression(),
)
label_error_indices = get_noise_indices(
    s = train_noisy_labels, 
    psx = probabilities, 
)

Some examples with FastText (NLP) and PyTorch (MNIST AlexNet).

Documentation: https://l7.curtisnorthcutt.com/cleanlab-python-package

I recognize that this is a stats forum and the expectation is a focus on mathematical derivations, but if it can be helpful and you're using Python, there is a package for classification with noisy labels called cleanlab: https://github.com/cgnorthcutt/cleanlab/. (Disclaimer: I am the author of cleanlab)

The cleanlab Python package, pip install cleanlab, for which I am an author, finds label errors in datasets and supports classification/learning with noisy labels. It works with scikit-learn, PyTorch, Tensorflow, FastText, etc.

For learning with noisy labels.

# Code taken from https://github.com/cgnorthcutt/cleanlab
from cleanlab.classification import LearningWithNoisyLabels
from sklearn.linear_model import LogisticRegression

# Learning with noisy labels in 3 lines of code.

# Wrap around any classifier. Works with sklearn/pyTorch/Tensorflow/FastText/etc.
lnl = LearningWithNoisyLabels(clf=LogisticRegression())
lnl.fit(X = X_train_data, s = train_noisy_labels)
# Estimate the predictions you would have gotten by training with *no* label errors.
predicted_test_labels = lnl.predict(X_test)

To find label errors in your dataset.

from cleanlab.latent_estimation import estimate_cv_predicted_probabilities

# Find the indices of label errors in 2 lines of code.

probabilities = estimate_cv_predicted_probabilities(
    X_train_data, 
    train_noisy_labels, 
    clf=LogisticRegression(),
)
label_error_indices = get_noise_indices(
    s = train_noisy_labels, 
    psx = probabilities, 
)

Some examples with FastText (NLP) and PyTorch (MNIST AlexNet).

Documentation: https://l7.curtisnorthcutt.com/cleanlab-python-package