0
$\begingroup$

When maximum likelihood is used to fit a parameter of a conditional distribution, I often see the argument go like this. We have our data set of experienced tuples $\mathcal{D} = \{(x_1,y_n), \cdots , (x_n, y_n)\}$. When we measure the conditional likelihood $$L(\theta) = P(y_1, \cdots, y_n \mid x_1, \cdots, x_n) \;,$$ we can decompose it as $$L(\theta) = L^{*}(\theta) = \prod_{i=1}^n P(y_i \mid x_i) \; ,$$ based off some assumptions. What are those assumptions, and can someone provide the full derivation of the likelihood from $L$ to $L^*$? For example, is $(x_i, y_i)$ assumed to be i.i.d for $i = \{1, \cdots ,n\}$?

$\endgroup$

2 Answers 2

1
$\begingroup$

I think I have a solution. We assume that $(x_i, y_i)$ is i.i.d. for $i = \{1, \cdots, n\}$. Then the joint likelihood $$L(\theta) = P((x_1, y_1), \cdots, (x_n, y_n)),$$ becomes $$L(\theta) = P\Big(x_1, y_1\Big)P\Big((x_2, y_2), \cdots, (x_n, y_n) \mid (x_1, y_1)\Big) = \cdots = \prod_{i = 1}^nP(x_i, y_i)$$ where I apply the i.i.d. assumption $n$ times. Then, when fitting the conditional distribution, we use Bayes rule $$L(\theta) = \prod_{i = 1}^nP(x_i)P(y_i \mid x_i),$$ and ignore the marginal distribution $p(x_i)$ when we are maximizing as we are only parameterizing $p(y_i \mid x_i)$.

My question is: is the i.i.d. assumption for $(x_i, y_i)$ where $i = \{1, \cdots, n\}$ typically the assumption used when doing conditional maximum likelihood?

$\endgroup$
1
$\begingroup$

The answer you gave to your question seems correct, although frankly I am not 100% sure about the "textbook" theory behind this.

Note that instead of saying "we are only parameterizing $p(y_i \mid x_i)$", you could be more explicit by including $\theta$ at all stages. You could even go one step further by replacing $$p(x,y)=p(y|x)p(x)$$ with $$p(x,y|\theta_X,\theta_Y) = p(y|x, \theta_X,\theta_Y) p(x|\theta_X,\theta_Y) = p(y|x, \theta_Y) p(x|\theta_X).$$ The second equality is based on the assumption that the distribution of $Y$ depends on parameter $\theta_Y$ (but not $\theta_X$), and similar for $X$. I believe this is an important assumption we can't avoid if we want the optimisation problem $$\max_{\theta_{X}}\prod_{i}p(x_{i}|\theta_{X})$$ to be equivalent to $$ \max_{\theta_{X}}\prod_{i}p(x_{i},y_{i}|\theta_{X},\theta_{Y})=\max_{\theta_{X}}\prod_{i}p(y_{i}|x_{i},\theta_{X},\theta_{Y})p(x_{i}|\theta_{X},\theta_{Y})$$ and necessary (that is, to be a sub-problem) for $$ \max_{\theta_{X},\theta_{Y}}\prod_{i}p(x_{i},y_{i}|\theta_{X},\theta_{Y})=\max_{\theta_{X},\theta_{Y}}\prod_{i}p(y_{i}|x_{i},\theta_{X},\theta_{Y})p(x_{i}|\theta_{X},\theta_{Y}).$$


I find it helpful to use the graphical models formalism. In terms of graphical models, our independence assumptions include $p(y|x, \theta_X,\theta_Y) = p(y|x, \theta_Y)$ and are described by the following diagram: $$ \begin{array}{ccc} \theta_{X} & & \theta_{Y}\\ \downarrow & & \downarrow\\ \left(X\right) & \to & \left(Y\right) \end{array} $$ In contrast, the following diagram permits that $p(y|x, \theta_X,\theta_Y) \neq p(y|x, \theta_Y)$: $$ \begin{array}{ccc} \theta_{X} & & \theta_{Y}\\ \downarrow & \searrow & \downarrow\\ \left(X\right) & \to & \left(Y\right) \end{array} $$ Here $\left(X\right),\left(Y\right)$ are random variables we sample, and $\theta_{X}, \theta_{Y}$ are parameters constant throughout our observation or experiment. That is, $(X_i,Y_i)$ are i.i.d. and parameterised by $\theta_{X}, \theta_{Y}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.