Logistic regression with labels corrupted by known noise model

Question

I am interested in knowing the "right way" to fit a binary logistic regression where the labels have been flipped with instance-specific noise probabilities that are known.

For the scenario I have in mind, we are give a dataset of triples, $$ D = \{ (x_i, y_i, q_i) \}_{i=1}^n$$ where $x_i \in \mathbb{R}^d$ are the feature vectors, $y_i \in \{0,1\}$ are noise corrupted labels, and $q_i \in [0, 1]$ are known corruption probabilities that are specific to each instance.

Given $x_i$ and $q_i$, the corrupted labels $y_i$ are generated from the following model:

• Let $p_i = \sigma(x_i^\top w + b)$ where $\sigma(\cdot)$ is the sigmoid function, and $w$ and $b$ are the the true but unknown logistic regression parameters.
• Let $z_i \in \{0,1\}$ be the uncorrupted label for $x_i$ where $z_i \sim \text{Bernoulli}(p_i)$.
• Finally the corrupted label $y_i$ is generated as$$ y_i = \begin{cases} z_i \text{ with probability } q_i \\ 1 - z_i \text{ with probability } 1 - q_i \\ \end{cases} $$

Given the above, it seems like the correct way to fit the logistic regression is to use the $q_i$ to create soft labels from the $y_i$ and minimize the KL-divergence to these soft labels.

However, I can alternatively imagine using the $q_i$ to weight each instance $(x_i, y_i)$ when fitting the model, so that loss is weighted more highly on instances whose corruption probabilities are low.

Any suggestions, insights or references about fitting logistic regression models in this scenario are very much welcomed.

Answer 1

I'm going to take a Bayesian approach because honestly I'm not sure how to tackle this as a Frequentist.

Let's first write out the likelihood for such a model. The likelihood of $y_i$ depends on a latent discrete parameter $z$ such that

$$ L(w, b \mid y, x) = \prod_i P(y_i \mid z_i)P(z_i \mid\beta) \>.$$

Here, each of the $P$ are a binomial likelihood. Since we don't observe the $z$, we need to integrate it out (or in this case, sum it out). Since the parameter is binary, this is fairly simple

$$ P(y_i \mid \beta) = P(y_i \mid z_i=0)P(z_i=0 \mid\beta) + P(y_i \mid z_i=1)P(z_i=1 \mid\beta) $$

I've done my best to simulate this process and write a Stan model which marginalizes out $z$. Here is the code to produce the simulation below

library(tidyverse)
library(cmdstanr)
library(posterior)
library(bayesplot)

set.seed(0)
n <- 10000
k <- 30
X <- cbind(1, MASS::mvrnorm(n, rep(0,k), 0.05*diag(k)))
b <-  c(-1.5, rnorm(k))
p <- boot::inv.logit(X%*%b)
z <- rbinom(n, 1, p)
q <- rbeta(n, 20, 120)
Q <- rbinom(n, 1, q)
y <- z*Q + (1-z)*(1-Q)

stan_code <- '
data{
  int n;
  int k;
  matrix[n, k+1] X;
  vector[n] q;
  array[n] int y;
}
parameters{
  vector[k+1] beta;
}
transformed parameters{
  vector[n] p = inv_logit(X*beta);
  vector[n] lik;
  
  for(i in 1:n){
    lik[i] = pow(q[i], y[i])*pow(1-q[i], 1-y[i])*p[i]; 
     // This is P(y|z=1)*p(z=1|beta)
    lik[i] += pow(1-q[i], y[i]) * pow(q[i], 1-y[i]) * (1-p[i]); 
     // This is P(y|z=0)*p(z=0|beta)
  }
}
model{
  beta ~ student_t(3.5, 0, 1);
  target += sum(log(lik));
}
generated quantities{
  array[n] int latent_z;
  array[n] int y_ppc;
  
  for(i in 1:n){
    latent_z[i] = bernoulli_rng(p[i]);
    y_ppc[i] = bernoulli_rng(q[i]) >0 ? latent_z[i] : 1-latent_z[i];
  }
}
'

model <- stan_code %>% 
         write_stan_file() %>% 
         cmdstan_model()

stan_data <- list(n=n, k=k, X=X, q=q, z=z, y=y)

fit <- model$sample(stan_data, parallel_chains = 4)

estimated_beta <- fit$draws('beta') %>% 
                  as_draws_matrix() %>% 
                  apply(., 2, mean)

plot(b, estimated_beta)
abline(0, 1)

bayesplot::ppc_bars(
  y, 
  as_draws_matrix(fit$draws('y_ppc'))
)

bayesplot::ppc_dens(as.numeric(p), as_draws_matrix(fit$draws('p')))

A few reasons to believe this is correct:

• The posterior expectations for the coefficients look similar

• Posterior predictive for $y$ looks good

• The distribution of probabilities for $z$ looks similar to what we simulated

EDIT:

In the stan model, I have written the likelihood and then incremented the target density by the log of the likelihood. This may be slightly less efficient than using log_sum_exp as follows

transformed parameters{
  vector[n] p = inv_logit(X*beta);
  vector[n] log_lik;
  
  for(i in 1:n){
    log_lik[i] = log_sum_exp(
      y[i]*log(q[i]) + (1-y[i])*log(1-q[i]) + log(p[i]),
      y[i]*log(1-q[i]) + (1-y[i])*log(q[i]) + log(1-p[i])
    );
  }
}

Now that we have the likelihood written out, it shouldn't be too hard to optimize it to get a frequentist approach.

Answer 2

(Started as a comment but got too long)

Some obvious references are: Label-noise robust logistic regression and its applications (2012) by Bootkrajang & Kaban and Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion (2019) by Tanno et al. That said, your setup seems to be akin to Estimating a Kernel Fisher Discriminant in the Presence of Label Noise (2001) where it is estimated simply by EM considering label noise being independent across different data-points and effectively we have a mixture of a normal distribution and a "probability table" (like yours).

I suppose one could also use a two-stage approach where they first do a soft-cluster assignment to all data and then treat the cluster assignment as weighting, but that seems messier.