2
$\begingroup$

From this slide: slides page 39, it says:

In probabilistic machine learning prediction time: compute the likelihood given parameters, each parameter configuration of which is weighted by the posterior: $$ p(y|x, D)=\int p(y|\theta, x) p(\theta|D) d\theta $$ where $x, y, \theta, D$ is input, output prediction, network parameters, dataset, respectively. I am trying to derive this formula in this way: $$ p(y|x, D)=\int p(y, \theta| x,D) d\theta = \int p(y|\theta, x, D) p(\theta|x, D) d\theta=\int p(y|\theta, x) p(\theta|D) d\theta $$ However, it depends on such assumption: $y$ is independent of $D$, so $p(y|\theta, x, D)=p(y|\theta, x)$. Since $y$ depends on $\theta$, $\theta$ depends on $D$, it seems that $y$ should also depends on $D$. What's wrong here?

In addition, we need $p(\theta|x, D)=p(\theta|D)$, that is $x$ is independent of $\theta$. Is it still correct in training time? It seems that $\theta$ depends on $D$, so should also depend on $x$?

$\endgroup$

1 Answer 1

1
$\begingroup$

The form with $p(y|\theta,x,D)$ is the correct version according to Bayes Rule. But, in the literature around these, typically we assume $p(y|\theta,x,D)=p(y|\theta,x)$ because $D$ is data useful in estimating the parameters of the model, which is $\theta$. However, if you already know the parameters of the model, you only need the input and the model parameters to calculate $y$'s distribution.

For the second term, same logic applies. We estimate $\theta$ from data, not the current value of $x$ at prediction time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.