Suppose I have two datasets, both of which consist of rows of features and their matching labels. One of these datasets is noise-free and its labels correspond to the ground truth, but the other is corrupted and some proportion of its labels are randomly flipped.

The question

How do we combine these datasets and account for their differing levels of noise? Presumably we shouldn't just throw out the noisy data, but it seems like we should also treat it differently from the higher quality noise-free data.

Should we modify the loss function? How should we construct the training/validation/test sets? Should we train two models separately and then combine their predictions?

My attempt at formalizing the problem

Let $\mathcal{D}$ and $\mathcal{D}_\text{noisy}$ be the noise-free and noisy datasets, respectively. For simplicity, assume we are attempting to learn a deterministic binary function $f(x) \in \{0, 1\}$. Then a single example from $\mathcal{D}$ or $\mathcal{D}_\text{noisy}$ is a feature/label pair $(x,y)$. For all $(x,y) \in \mathcal{D}$, we have $y = f(x)$. But if $(x,y)$ comes from $\mathcal{D}_\text{noisy}$, then $$ y = \begin{cases} f(x) & \text{with probability }p \\ 1 - f(x) & \text{with probability }1 - p, \\ \end{cases} $$ where $p$ is known.

How do we combine $\mathcal{D}$ and $\mathcal{D}_\text{noisy}$, and how do we factor in the noise parameter $(1-p)$?

  • $\begingroup$ It sounds like you want a noisy data detector. You need to have inputs to a function. Something about the nature of the noisy vs not-noisy must be able to be extracted from that using functions you can use. Then you can start making progress moving forward. $\endgroup$ Jan 5, 2021 at 17:01


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.