Adjusting model output when the target is noisy

Question

Imagine we have a binary classification model: predicting if a person $x$ is F (female) or M (male). However, the target variable that we observe (say, $y$) is noisy. So there is some other variable that we don't observe with the true value (say, $g$)

We don't know $g$, but we know the conditional distribution $P(g \mid y)$:

• $P(g = F \mid y = F) = 0.76$, $P(g = M \mid y = F) = 0.24$
• $P(g = F \mid y = M) = 0.20$, $P(g = M \mid y = M) = 0.80$

Additionally, we know the marginal distribution of $y$:

• $P(y = F) = 1/3$, $P(y = M) = 2/3$

Now we train a model $P(y \mid x)$ on these $y$ (say, logistic regression).

How can we correct the output of the model to account for the noise in the data? That is, how do we get $P(g \mid x)$ from $P(y \mid x)$ using $P(y)$ and $P(g \mid y)$?

Answer 1

As I understand the problem, the following joint distribution is given: \begin{equation} p(g,y|x) = p(g|y)\,p(y|x) . \end{equation} The right-hand side embodies the assumption $p(g|x,y) = p(g|y)$. It follows immediately that \begin{equation} p(g|x) = \int p(g|y)\,p(y|x)\,dy . \end{equation} In the binary case, this integral can be expressed as a sum: \begin{equation} p(g=F|x) = p(g=F|y=F)\,p(y=F|x) + p(g=F|y=M)\,p(y=M|x) , \end{equation} with a similar expression for $p(g=M|x)$.

Perhaps I have misunderstood the problem, because I don't see any role for $p(y)$.

Answer 2

What you describe sounds like a state space model(SSM). Look at Kalman filter, which is the estimation approach for a specific case of SSM with Gaussian errors.

In a (simplified) Kalman filter you have: $$ g_k = \mathbf{F}_k g_{k-1} + \mathbf{B}_k x_{k} + w_k $$ $$y_k = \mathbf{H}_k y_k + v_k$$ where $w,v$ - are errors.

Obviously, it's not exactly your problem because your errors are not Gaussian, and the dependent variable is discrete. However, there are discrete state space models, and I'm sure you can figure out how to apply them to your case.

The key here is that there's an observable state equation for $g$ and observable measurement equation for $y$. So the Kalman filter has two passes: filter and smoothing. Roughly, in the filtering pass you predict the next $y$, then in smoothing stage you estimate the $g$ including the information on discrepancy between the prediction and actual.

Answer 3

The problem was nicely described by @mef in another answer. It can be summarised by this graph:

In the graph, the first layer is your model for $P(y | x)$, and the second layer is your model for the relationship between $y$ and $g$, i.e. $P(g|y)$. In the end, you can get a model for the gender of each person:

\begin{align} P(g = M | x) &= \sum _{S = \{ M, F \}} P(g = M|y = S) \, \cdot \, P(y = S | x) \\ P(g = F | x) &= \sum _{S = \{ M, F \}} P(g = F|y = S) \, \cdot \, P(y = S | x) \end{align}

Filling in the model for $g$ given $y$,

\begin{align} P(g = M | x) &= 0.8 - 0.56 \cdot P(y = F | x) \\ P(g = F | x) &= 0.2 + 0.56 \cdot P(y = F | x) \end{align}