Deriving conditional maximum likelihood

Question

When maximum likelihood is used to fit a parameter of a conditional distribution, I often see the argument go like this. We have our data set of experienced tuples $\mathcal{D} = \{(x_1,y_n), \cdots , (x_n, y_n)\}$. When we measure the conditional likelihood $$L(\theta) = P(y_1, \cdots, y_n \mid x_1, \cdots, x_n) \;,$$ we can decompose it as $$L(\theta) = L^{*}(\theta) = \prod_{i=1}^n P(y_i \mid x_i) \; ,$$ based off some assumptions. What are those assumptions, and can someone provide the full derivation of the likelihood from $L$ to $L^*$? For example, is $(x_i, y_i)$ assumed to be i.i.d for $i = \{1, \cdots ,n\}$?

Answer 1

I think I have a solution. We assume that $(x_i, y_i)$ is i.i.d. for $i = \{1, \cdots, n\}$. Then the joint likelihood $$L(\theta) = P((x_1, y_1), \cdots, (x_n, y_n)),$$ becomes $$L(\theta) = P\Big(x_1, y_1\Big)P\Big((x_2, y_2), \cdots, (x_n, y_n) \mid (x_1, y_1)\Big) = \cdots = \prod_{i = 1}^nP(x_i, y_i)$$ where I apply the i.i.d. assumption $n$ times. Then, when fitting the conditional distribution, we use Bayes rule $$L(\theta) = \prod_{i = 1}^nP(x_i)P(y_i \mid x_i),$$ and ignore the marginal distribution $p(x_i)$ when we are maximizing as we are only parameterizing $p(y_i \mid x_i)$.

My question is: is the i.i.d. assumption for $(x_i, y_i)$ where $i = \{1, \cdots, n\}$ typically the assumption used when doing conditional maximum likelihood?

Answer 2

The answer you gave to your question seems correct, although frankly I am not 100% sure about the "textbook" theory behind this.

Note that instead of saying "we are only parameterizing $p(y_i \mid x_i)$", you could be more explicit by including $\theta$ at all stages. You could even go one step further by replacing $$p(x,y)=p(y|x)p(x)$$ with $$p(x,y|\theta_X,\theta_Y) = p(y|x, \theta_X,\theta_Y) p(x|\theta_X,\theta_Y) = p(y|x, \theta_Y) p(x|\theta_X).$$ The second equality is based on the assumption that the distribution of $Y$ depends on parameter $\theta_Y$ (but not $\theta_X$), and similar for $X$. I believe this is an important assumption we can't avoid if we want the optimisation problem $$\max_{\theta_{X}}\prod_{i}p(x_{i}|\theta_{X})$$ to be equivalent to $$ \max_{\theta_{X}}\prod_{i}p(x_{i},y_{i}|\theta_{X},\theta_{Y})=\max_{\theta_{X}}\prod_{i}p(y_{i}|x_{i},\theta_{X},\theta_{Y})p(x_{i}|\theta_{X},\theta_{Y})$$ and necessary (that is, to be a sub-problem) for $$ \max_{\theta_{X},\theta_{Y}}\prod_{i}p(x_{i},y_{i}|\theta_{X},\theta_{Y})=\max_{\theta_{X},\theta_{Y}}\prod_{i}p(y_{i}|x_{i},\theta_{X},\theta_{Y})p(x_{i}|\theta_{X},\theta_{Y}).$$

---

I find it helpful to use the graphical models formalism. In terms of graphical models, our independence assumptions include $p(y|x, \theta_X,\theta_Y) = p(y|x, \theta_Y)$ and are described by the following diagram: $$ \begin{array}{ccc} \theta_{X} & & \theta_{Y}\\ \downarrow & & \downarrow\\ \left(X\right) & \to & \left(Y\right) \end{array} $$ In contrast, the following diagram permits that $p(y|x, \theta_X,\theta_Y) \neq p(y|x, \theta_Y)$: $$ \begin{array}{ccc} \theta_{X} & & \theta_{Y}\\ \downarrow & \searrow & \downarrow\\ \left(X\right) & \to & \left(Y\right) \end{array} $$ Here $\left(X\right),\left(Y\right)$ are random variables we sample, and $\theta_{X}, \theta_{Y}$ are parameters constant throughout our observation or experiment. That is, $(X_i,Y_i)$ are i.i.d. and parameterised by $\theta_{X}, \theta_{Y}$.