I know from Bayes theorem we have the formula "posterior $\propto$ prior * likelihood" or more concrete $Pr(\theta|D) \propto Pr(\theta)Pr(D|\theta)$ with $Pr(\theta)$ is prior distribution of parameter, and $D$ is observed data. But in a book I read when apply it to the following situation, the result confuses me.

Situation: We want to approximate a real function $f: X = R^k \rightarrow R$ by a linear function $h(x,w)$, $w$ is the parameter vector. Having observed data $D = (X,Y) = (x_1,y_1),...,(x_n,y_n)$, $y_i$ is label of $x_i$.

Now let $Pr(w)$ is prior distribution of parameter $w$. In the book, have result that is

$$Pr(w|X,Y)\propto Pr(w)Pr(Y|X,w))\quad (1)$$ But from my understand, by applying $Pr(\theta|D) \propto Pr(\theta)Pr(D|\theta)$ , I think

$$Pr(w|X,Y)\propto Pr(w)Pr(X,Y|w)\quad (2)$$ The difference between (1) and (2) is $Pr(Y|X,w)$ and $Pr(X,Y|w)$.

Could you tell me why we have $(1)$?

You may want to know what the book is? But it is not in english, and there is not ebook form for it, so I don't post it here.

I try to test if (1) and (2) are the same? I transform $Pr(X,Y|w)$ as follows:

$$Pr(X,Y|w)=Pr(X,Y,w)/Pr(w)=Pr(Y|X,w)Pr(X,w)/Pr(w)=Pr(Y|X,w)Pr(w)Pr(X|w)/Pr(w)=Pr(Y|X,w)Pr(X|w)$$ So $(1)$ & $(2)$ are not the same.

  • $\begingroup$ Are the X's being conditioned on, as with a regression model? $\endgroup$ – Glen_b Jun 15 '17 at 6:34
  • $\begingroup$ No. But what are the conditions that you expect? Is that conditions make answer clearer $\endgroup$ – ydhhat Jun 15 '17 at 7:51

This situation is that we want to know how to predict $Y$ when we know $X$, which means learning about $w$. So the RHS of the equation should not contain a $p(X|whatever)$ term, because we are interested in cases where $X$ is fixed already.

In terms of the algebra:




This does assume that we are happy with writing $p(X)$.

*I suspect the use of the $\propto$ convention here makes the derivation less clear because it sweeps terms that give you a clue how to proceeed under the carpet.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.