I have seen the general form of posterior for a regression $y = f(x)$ defined as $$ P(\theta|y,x) = \frac{ P(y | x, \theta) P(\theta) }{ P(y|x) } $$

I would like to know how to arrive at this form, starting from Bayes rule and the laws of probability.

The approach I tried is to write Bayes theorem notated as $p(\theta|v) = \frac{ p(v|\theta) p(\theta) }{ p(v) }$ and then try subsituting something for $v$.

Approach A. With $v \rightarrow y|x$, this gives $$ " P(\theta|y|x) = \frac{ P(y|x | \theta) P(\theta) }{ P(y|x) } " $$ and then if there is a rule that $a|b|c \rightarrow a|b,c$ it gives the result. However I have not seen such a rule. Does it exist?

Approach B. Start with $v \rightarrow y$ giving $$ P(\theta|y) = \frac{ P(y | \theta) P(\theta) }{ P(y) } $$ and then assume there is a rule that you can condition every factor on some other variable $x$ (see rule below), giving $$ P(\theta|y,x) = \frac{ P(y | \theta,x) P(\theta|x) }{ P(y|x) } $$ and lastly assume that $P(\theta|x) = P(\theta)$. But here I have not seen a rule that allows $$ \text{if} \quad p(A) = P(B)P(C)\cdots \quad\text{then}\quad p(A|X) = P(B|X)P(C|X)\cdots $$ Does such a rule exist?

What is the right approach?

  • 3
    $\begingroup$ To write that $\nu=y|x$ does not make sense: $y$ is a random variable that has both a marginal distribution (if irrelevant here) and a conditional distribution given the random variable $x$. The notation $P(y|x|\theta)$ does not make sense either. $\endgroup$
    – Xi'an
    Jan 2 '19 at 7:53

\begin{align*} P(\theta \mid y,x) &= \frac{P(\theta, y,x)}{P(y,x)} \tag{defn. condtl. prob} \\ &= \frac{P(y \mid \theta, x)P(\theta \mid x)P(x)}{P(y \mid x)P(x)} \tag{defn. condtl. prob}\\ &= \frac{P(y \mid \theta, x)P(\theta \mid x)}{P(y \mid x)} \tag{cancellation}\\ &= \frac{ P(y | x, \theta) P(\theta) }{ P(y|x) } \tag{indep.} \end{align*}


Your initial idea is correct, but you should have used $x,y$ as $v$. Then $p(v)=p(x,y)=p(y|x)p(x)$, and:

$$ p(\theta\vert v)=p(\theta\vert x,y) =\frac{p(v\vert\theta)p(\theta)}{p(v)} =\frac{p(y\vert x,\theta)p(x\vert\theta)p(\theta)}{p(y\vert x)p(x)}=\frac{p(y\vert x,\theta)p(\theta)}{p(y\vert x)} $$

Where we used the fact that $p(x\vert\theta)=p(x)$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.