3
$\begingroup$

I am revising Bayesian linear regression from Bishop's book and the following material. In page 275 it provides the formula for the posterior distribution, which, in Bishop's book is the following:

$$ \underbrace{p(\theta|D)}_{\text{posterior}} =\frac{\overbrace{p(\theta)}^{\text{prior}}\overbrace{p(D|\theta)}^{\text{likelihood}}}{\underbrace{p(D)}_{\text{evidence}}}=\frac{\overbrace{p(\theta)}^{\text{prior}}\overbrace{p(D|\theta)}^{\text{likelihood}}}{\underbrace{\int p(\theta)p(D|\theta)d\theta}_{\text{evidence}}} $$

which D represents our data. In the hyperlinked material the same equation it is phrased as:

$$p(\mathbf \theta |Y,X) = \frac{p(Y|X,\mathbf{\theta}) p(\mathbf{\theta})}{p(Y|X)}$$

where $\mathbf{x}_{n} \in \mathbb{R}^{D}$ is the one of the vector for the matrix $X$ with the correspondent label to be $y_{n} \in \mathbb{R}$ and $\mathbf{\theta}$ the regression parameters. I guess roughly $D = \{X, Y\}$.

My question is that I am struggling to understand the likelihood in the second equation. Why is it $p(Y|X,\mathbf{\theta})$ and not $p(Y, X| \mathbf{\theta}$). The same for the evidence. Also what exactly is the interpretation of $p(Y|X)$ and $p(X|Y)$.

$\endgroup$
1
$\begingroup$

Bayes rule is a way of "inverting" any joint distribution, even one conditioned on other things. Consider the joint distribution for $A$ and $B$ conditioned on $C$, which can be factored two ways: \begin{equation} \begin{split} p(A,B|C) &= p(A|B,C)\,p(B|C) \\ &= p(B|A,C)\,p(A|C) . \end{split} \end{equation} One way of applying Bayes rule to the two factorizations delivers \begin{equation} p(A|B,C) = \frac{p(B|A,C)\,p(A|C)}{p(B|C)} , \end{equation} where \begin{equation} p(B|C) = \int p(B|A,C)\,p(A|C)\,dA . \end{equation}

To accommodate the first example, we can let $A = \theta$, $B=D$, and (because there is no conditioning with respect to the joint distribution in the first example) $C = \varnothing$ [the empty set].

To accommodate the second example, we can let $A = \theta$, $B = Y$, and $C = X$.

The upshot is this: where a particular thing appears in Bayes rule is not determined by whether or not the thing is "data", but whether or not it is being conditioned on in the joint distribuiton.

But we are not done with the second example because something doesn't match. The second example has $p(\theta)$ but our use of the general statement produces $p(\theta|X)$. What's going on here? The answer is that the second example has a special assumption built into it: \begin{equation} p(\theta|X) = p(\theta) . \end{equation} In other words, the prior for $\theta$ doesn't depend on the "covariates" $X$. This assumption is one that is usually made, so that's not what makes it "special." The assumption is special because it asserts something beyond what Bayes rule by itself says.

Thus far I've addressed the questions about the form of the likelihood and of the evidence. Once one accepts these forms, I'm not sure what there is to say of the interpretation of $p(Y|X)$.

$\endgroup$
2
  • $\begingroup$ That is surely a very nice answer I have to admit. Thanks a lot. $\endgroup$
    – Jose Ramon
    Aug 29 '21 at 9:53
  • $\begingroup$ For the last part of my question, I was wondering more about the verbal meaning of the conditions in the case of the ML setting. So for instance $P(X|Y)$ means sth like given that we observe labels $Y$ what is the probability to have a specific type of data $X$? Or in the case of $P(X|\theta)$: given the parameters $\theta$ what is the probability observing data $X$. $\endgroup$
    – Jose Ramon
    Aug 29 '21 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.