I was trying to understand Laplace approximation in statistics and so I was going through the wikipedia article. I don't know much about statistics and I am already getting a bit confused by the notations. It is stated $$p(y,\theta|x)=p(y|x,\theta)p(\theta)=p(y|x)p(\theta|y,x).$$ However when using definitions, I get $$p(y,\theta|x)=\frac{p(y,\theta,x)}{p(x)}=\frac{p(y,\theta,x)}{p(x,\theta)}\cdot \frac{p(x,\theta)}{p(x)}=p(y|x,\theta)p(\theta| x)\qquad (1)$$ first I wonder why the "given $x$" part has been omitted.

Second, I saw that when we have the data $\mathcal{D}$, bayesian inference is usually denoted $$p(\theta|\mathcal{D})=\frac{p(\mathcal{D}|\theta)}{p(\mathcal{D})}p(\theta)$$ and since the data are here $\mathcal{D}=\{(x_i,y_i)\}_{i=1}^n$, I was wondering why instead of (1) we don't have, $$p(\theta|(x,y))=\frac{p((x,y)|\theta)}{p((x,y))}p(\theta).$$

Third, it is said

Laplace's approximation provides an analytical expression for a posterior probability distribution by fitting a Gaussian distribution with a mean equal to the MAP solution and precision equal to the observed Fisher information.

and in the article they proceed by computing this mean of the Gaussian by $\hat{\theta}=\underset{\theta}{argmax}\operatorname{log} p(y,\theta|x)$. However, looking at the wikipedia for MAP, it seems to me the mean they compute is rather the MLE estimator and the MAP estimator would be $\hat{\theta}=\underset{\theta}{argmax}\operatorname{log} p(\theta|x,y)$. I would be glad if I could get some clarifications on these matters.

  • $\begingroup$ The parameters are assumed to be independent of $x$, so $p(\theta \mid x) \triangleq p(\theta)$. $\endgroup$ Jan 24 at 9:23
  • $\begingroup$ @AryaMcCarthy, but then why $\cdot | x$ is present in the other terms? It seems inconsistent $\endgroup$
    – roi_saumon
    Jan 24 at 10:01
  • $\begingroup$ Because you can't remove it from the other terms. The purpose of the derivation is to represent $p(y,\theta|x)$. In the other terms, there is no assumed conditional independence. $\endgroup$ Jan 24 at 12:22
  • $\begingroup$ Anyways, for (2), these are identical and just a matter of notation. For (3), I agree this is an error and should be posterior. One of the references on the Wiki page is a little vague about it <inference.org.uk/mackay/itprnn/ps/341.342.pdf>, but does mention the expansion should be about the peak of the probability distribution of interest, which is indeed the posterior, and not the likelihood (which is not even necessarily interpretable as a distribution). $\endgroup$ Jan 24 at 15:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.