Overview
Suppose I have two datasets, both of which consist of rows of features and their matching labels. One of these datasets is noise-free and its labels correspond to the ground truth, but the other is corrupted and some proportion of its labels are randomly flipped.
The question
How do we combine these datasets and account for their differing levels of noise? Presumably we shouldn't just throw out the noisy data, but it seems like we should also treat it differently from the higher quality noise-free data.
Should we modify the loss function? How should we construct the training/validation/test sets? Should we train two models separately and then combine their predictions?
My attempt at formalizing the problem
Let $\mathcal{D}$ and $\mathcal{D}_\text{noisy}$ be the noise-free and noisy datasets, respectively. For simplicity, assume we are attempting to learn a deterministic binary function $f(x) \in \{0, 1\}$. Then a single example from $\mathcal{D}$ or $\mathcal{D}_\text{noisy}$ is a feature/label pair $(x,y)$. For all $(x,y) \in \mathcal{D}$, we have $y = f(x)$. But if $(x,y)$ comes from $\mathcal{D}_\text{noisy}$, then $$ y = \begin{cases} f(x) & \text{with probability }p \\ 1 - f(x) & \text{with probability }1 - p, \\ \end{cases} $$ where $p$ is known.
How do we combine $\mathcal{D}$ and $\mathcal{D}_\text{noisy}$, and how do we factor in the noise parameter $(1-p)$?
