# Adjusting model output when the target is noisy

Imagine we have a binary classification model: predicting if a person $x$ is F (female) or M (male). However, the target variable that we observe (say, $y$) is noisy. So there is some other variable that we don't observe with the true value (say, $g$)

We don't know $g$, but we know the conditional distribution $P(g \mid y)$:

• $P(g = F \mid y = F) = 0.76$, $P(g = M \mid y = F) = 0.24$
• $P(g = F \mid y = M) = 0.20$, $P(g = M \mid y = M) = 0.80$

Additionally, we know the marginal distribution of $y$:

• $P(y = F) = 1/3$, $P(y = M) = 2/3$

Now we train a model $P(y \mid x)$ on these $y$ (say, logistic regression).

How can we correct the output of the model to account for the noise in the data? That is, how do we get $P(g \mid x)$ from $P(y \mid x)$ using $P(y)$ and $P(g \mid y)$?

• As I understand, $y$ is "observed sex" (noisy measure) and $g$ is actual sex. So what is $x$? – LmnICE Sep 13 '17 at 10:30
• $x$ is the person - i.e. the features that describe them – Alexey Grigorev Sep 13 '17 at 13:53
• So the model predicts $y$ from $x$, only $y$ is a noisy measure of $g$? Why not predict $g$ directly? – LmnICE Sep 13 '17 at 14:05
• Because $g$ is not available, only $P(g|y)$ is – Alexey Grigorev Sep 13 '17 at 14:07

As I understand the problem, the following joint distribution is given: $$p(g,y|x) = p(g|y)\,p(y|x) .$$ The right-hand side embodies the assumption $p(g|x,y) = p(g|y)$. It follows immediately that $$p(g|x) = \int p(g|y)\,p(y|x)\,dy .$$ In the binary case, this integral can be expressed as a sum: $$p(g=F|x) = p(g=F|y=F)\,p(y=F|x) + p(g=F|y=M)\,p(y=M|x) ,$$ with a similar expression for $p(g=M|x)$.

Perhaps I have misunderstood the problem, because I don't see any role for $p(y)$.

• probably the model (like logreg) will account for $p(y)$ – Alexey Grigorev Sep 14 '17 at 7:55

What you describe sounds like a state space model(SSM). Look at Kalman filter, which is the estimation approach for a specific case of SSM with Gaussian errors.

In a (simplified) Kalman filter you have: $$g_k = \mathbf{F}_k g_{k-1} + \mathbf{B}_k x_{k} + w_k$$ $$y_k = \mathbf{H}_k y_k + v_k$$ where $w,v$ - are errors.

Obviously, it's not exactly your problem because your errors are not Gaussian, and the dependent variable is discrete. However, there are discrete state space models, and I'm sure you can figure out how to apply them to your case.

The key here is that there's an observable state equation for $g$ and observable measurement equation for $y$. So the Kalman filter has two passes: filter and smoothing. Roughly, in the filtering pass you predict the next $y$, then in smoothing stage you estimate the $g$ including the information on discrepancy between the prediction and actual.

The problem was nicely described by @mef in another answer. It can be summarised by this graph:

In the graph, the first layer is your model for $P(y | x)$, and the second layer is your model for the relationship between $y$ and $g$, i.e. $P(g|y)$. In the end, you can get a model for the gender of each person:

\begin{align} P(g = M | x) &= \sum _{S = \{ M, F \}} P(g = M|y = S) \, \cdot \, P(y = S | x) \\ P(g = F | x) &= \sum _{S = \{ M, F \}} P(g = F|y = S) \, \cdot \, P(y = S | x) \end{align}

Filling in the model for $g$ given $y$,

\begin{align} P(g = M | x) &= 0.8 - 0.56 \cdot P(y = F | x) \\ P(g = F | x) &= 0.2 + 0.56 \cdot P(y = F | x) \end{align}