Consider a set of $m$ examples $X=\{x^{(1)},x^{(2)},\cdots, x^{(m)}\}$ drawn independently from the true but unknown data-generating distribution $p_{\text{data}}(x)$.

Let $p_{\text{model}}$ be a parametric family of probability distributions over the same space indexed by $\theta$.

The likelihood function is defined as $L(\theta|X) =p_{\text{model}}(x^{(1)}, x^{(2)},\cdots,x^{(m)};\theta)$.

Because of the independence assumption, we can write $L(\theta|X)=\prod_{i=1}^m p_{\text{model}}(x^{(i)};\theta)$

Now, suppose we are $m$ examples $\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\cdots, (x^{(m)},y^{(m)})\}$ and we are want to estimate the conditional probability $P(y|x;\theta)$ using the conditional maximum likelihood estimation, so that we can predict $y$ given $x$ .

How do I define the conditional likelihood? The problem I'am facing is that we cannot write $L(\theta; y|x)=P(Y|X;\theta)$ since $X$ contains different $x^{(i)}$s. But I think it makes sense to 'define' $L(\theta;y|x)=\prod_{i=1}^mP(y^{(i)}|x^{(i)};\theta)$ although in unconditional likelihood this followed from the definition of likelihood and the independence assumption. But I don't think I am comfortable with this. How does one define conditional likelihood?


Usually one assumes that there is a distribution $$p_{\text{data}}(y,x)$$ that somewhat defines not only the distributions of $x$ and $y$ but also their dependency (i.e. if $y_i = f(x_i)$ then we can estimate $f$ by computing the conditional expectation $E[y|X=x]$ with respect to this common probability distribution and so on). Now we do not only assume that the $x_i$ were drawn independently but rather that the whole tuples $(y_1, x_1), ..., (y_n,x_n)$ were drawn independently. Caution: this does in no way mean that $y_i$ is somewhat independent from $x_i$, it only means that $$p(y,x) = \prod_{i=1}^n p(y_i, x_i)$$ and by using marginalization and the Theorem of Fubini (wikipedia) we see that \begin{align*} p(y|x) &= \frac{p(y,x)}{p(x)} = \frac{p(y,x)}{\int p(\hat{y}, x) d\hat{y}} \\ &= \frac{p(y,x)}{\int ... \int \prod_{i=1}^n p(\hat{y}_i, x) d\hat{y}_1 ... d\hat{y}_n} \\ &= \frac{\prod_{i=1}^n p(y_i,x_i)}{\prod_{i=1}^n \int p(\hat{y}_i, x) d\hat{y}_i} \\ &= \prod_{i=1}^n p(y_i|x_i) \end{align*}

So, you can safely feel comfortable with this, it follows from the basic assumptions we always have: the observed data are independent.

Edit: note that usually we do not make assumptions about the joint probability $p(y,x)$ or $p(y_i, x_i)$ directly but we rather assume that $y_i = f(x_i) + \text{'small' error}$ for a single function $f$ and then we make assumptions on $f$, for example, in linear regression we assume that $$f(x) = \beta^T \cdot x$$

| cite | improve this answer | |
  • $\begingroup$ How do you derive the last step from the penultimate? $\endgroup$ – nbro Nov 1 '19 at 20:27
  • $\begingroup$ @nbro: what exactly do you mean? Is it about $\frac{\prod_i p(y_i,x_i)}{\prod_i \int ...} = \prod_i p(y_i|x_i)$? $\endgroup$ – Fabian Werner Nov 2 '19 at 18:33
  • $\begingroup$ Yes, I mean that step. I realize that's just the Bayes theorem, but with a product outside, but can you be more explicit about how the basic Bayes theorem generalizes to this case? $\endgroup$ – nbro Nov 2 '19 at 23:45
  • $\begingroup$ I do not understand, for any pair of random variables $X, Y$ we define (i.e. no theorem involved) $p(x|y) = p(x,y)/p(y)$. Now for each factor $p(y_i,x_i)/\int_{\hat{y}_i} p(\hat{y}_i, x_i) d\hat{y}_i$ the denominator is just $p(x_i)$ by marginalization, i.e. the $i$-th factor is nothing else than $p(y_i,x_i)/p(x_i) = p(y_i|x_i)$ by definition (not by theorem)... $\endgroup$ – Fabian Werner Nov 4 '19 at 10:02
  • 1
    $\begingroup$ @nbro: Let us take this offline to chat: chat.stackexchange.com/rooms/100787/… $\endgroup$ – Fabian Werner Nov 7 '19 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.