0
$\begingroup$

When maximum likelihood is used to fit a parameter of a conditional distribution, I often see the argument go like this. We have our data set of experienced tuples $\mathcal{D} = \{(x_1,y_n), \cdots , (x_n, y_n)\}$. When we measure the conditional likelihood $$L(\theta) = P(y_1, \cdots, y_n \mid x_1, \cdots, x_n) \;,$$ we can decompose it as $$L(\theta) = L^{*}(\theta) = \prod_{i=1}^n P(y_i \mid x_i) \; ,$$ based off some assumptions. What are those assumptions, and can someone provide the full derivation of the likelihood from $L$ to $L^*$? For example, is $(x_i, y_i)$ assumed to be i.i.d for $i = \{1, \cdots ,n\}$?

$\endgroup$
0
$\begingroup$

I think I have a solution. We assume that $(x_i, y_i)$ is i.i.d. for $i = \{1, \cdots, n\}$. Then the joint likelihood $$L(\theta) = P((x_1, y_1), \cdots, (x_n, y_n)),$$ becomes $$L(\theta) = P\Big(x_1, y_1\Big)P\Big((x_2, y_2), \cdots, (x_n, y_n) \mid (x_1, y_1)\Big) = \cdots = \prod_{i = 1}^nP(x_i, y_i)$$ where I apply the i.i.d. assumption $n$ times. Then, when fitting the conditional distribution, we use Bayes rule $$L(\theta) = \prod_{i = 1}^nP(x_i)P(y_i \mid x_i),$$ and ignore the marginal distribution $p(x_i)$ when we are maximizing as we are only parameterizing $p(y_i \mid x_i)$.

My question is: is the i.i.d. assumption for $(x_i, y_i)$ where $i = \{1, \cdots, n\}$ typically the assumption used when doing conditional maximum likelihood?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.