I was reading this paper http://stat.wharton.upenn.edu/~lzhao/papers/MyPublication/MultiExpert_ICML_2009.pdf

However, I have some confusion related to maximum likelihood derivation in this paper

We have the parameters of the model

$\theta = \{w,\alpha,\beta\}$

The maximum likelihood is given by

$P(D|\theta) = \prod_{i=1}^{i=N}P(y_i^{1},...y_i^{R}|x_i,\theta)$

where $y_i^1,...y_i^R$ are the noisy labels predicted by my R predictors.

Then the paper says conditioning on the true label, $y_i$ and also using the assumption that $\alpha^j$ and $\beta^j$ do not depend on the instance $x_i$ the likelihood can be written as

$P(D|\theta) = \prod_{i=1}^{i=N}P[y_i^1...y_i^R|y_i=1,\alpha].P[y_i=1,x_i,w] + P[y_i^1...y_i^R|y_i=0,\beta].P[y_i=0,x_i,w]$

I didn't get how this equation was derived. Any insights will be helpful.


The likelihood (not the maximum likelihood as you've written) is given by $$P\left[\mathcal{D}\mid\theta\right]=\prod_{i=1}^{N}P\left[y_i^1,\ldots,y_i^R\mid x_i,\theta\right]$$ This can be rewritten as the sum of the probabilities of two disjoint events.

$$P\left[\mathcal{D}\mid\theta\right]=\prod_{i=1}^{N}\left(P\left[y_i^1,\ldots,y_i^R,(y=0)\mid x_i,\theta\right]+P\left[y_i^1,\ldots,y_i^R,(y=1)\mid x_i,\theta\right]\right)$$

Using conditional probability we get that

\begin{eqnarray*} P\left[\mathcal{D}\mid\theta\right]&=&\prod_{i=1}^{N}\left(P\left[y_i^1,\ldots,y_i^R\mid x_i,\theta,(y=0)\right]P\left[y=0\mid x_i,\theta\right]+\right.\\ & & \left. P\left[y_i^1,\ldots,y_i^R\mid x_i,\theta,(y=1)\right]P\left[y=1\mid x_i,\theta\right]\right) \end{eqnarray*}

Once we condition on the event that $y$ is a certain value, each of the $y_i^j$ becomes a Bernoulli random variable with parameter either $\alpha^j$ (if $y=1$) or $\beta^j$ (if $y=0$); the $x_i$ and other parameters from $\theta$ do not affect the likelihood. This gives us that

\begin{eqnarray*} P\left[\mathcal{D}\mid\theta\right]&=&\prod_{i=1}^{N}\left(P\left[y_i^1,\ldots,y_i^R\mid \beta,(y=0)\right]P\left[y=0\mid x_i,\theta\right]+\right.\\ & & \left. P\left[y_i^1,\ldots,y_i^R\mid \alpha,(y=1)\right]P\left[y=1\mid x_i,\theta\right]\right) \end{eqnarray*}

Now we are also assuming that the probability that $y$ takes on a certain value depends only on $x$ and $w$, and no other parameters from $\theta$.

\begin{eqnarray*} P\left[\mathcal{D}\mid\theta\right]&=&\prod_{i=1}^{N}\left(P\left[y_i^1,\ldots,y_i^R\mid \beta,(y=0)\right]P\left[y=0\mid x_i,w\right]+\right.\\ & & \left. P\left[y_i^1,\ldots,y_i^R\mid \alpha,(y=1)\right]P\left[y=1\mid x_i,w\right]\right) \end{eqnarray*}

The important equations from the paper are $(3)-(5)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.