3
$\begingroup$

I have a model where the noise is modelled as independent and identically distributed across the various data points. The noise $e$ is modelled as a 0 mean gaussian with $\phi$ as the precision (inverse variance). The likelihood for a single data point $i$ is defined as:

$$ P(y|w, \phi) = (\frac{\phi}{2\pi})^{0.5} \exp ^{-0.5 e_{i}\phi e_{i}} $$

The prior $P(w|y)$ is also modelled as a multivariate gaussian with zero mean and a covariance matrix. There is a scale parameter $\lambda$ which is also defined probabilistically and modelled using a gamma distribution. The noise precision $\phi$ also needs be inferred and is modelled using a gamma distribution

So ultimately the posterior will be given us

$$ P(w, \phi, \lambda|y) = \frac{P(w|\lambda) p(\phi) p(\lambda) \prod_{i} P(y|w, \phi)}{P(y)} $$

Now, this is probably a really silly question but since the priors are modelled using the Gamma or Gaussian distribution and the likelihood is also modelled with Gaussian noise, can we say that the posterior distribution will have also belong, say to the exponential family of distributions? I would like to know how the shape of priors and the likelihood affect the shape of the posterior distribution, if at all.

$\endgroup$
  • $\begingroup$ "The noise e is modelled as a 0 mean gaussian with ϕ as the precision"..."The noise variance ϕ also needs be inferred and is modelled using a gamma distribution". Is $\phi$ the noise precision or variance? $\endgroup$ – Zhubarb Jan 2 '14 at 10:09
  • $\begingroup$ Sorry it is the precision. Thanks for catching that. Fixed. $\endgroup$ – Luca Jan 2 '14 at 10:27
2
$\begingroup$

I would like to know how the shape of priors and the likelihood affect the shape of the posterior distribution, if at all.

The posterior is proportional to the product of the likelihood and the prior:

$$p(\theta |\textrm{Data})=\frac{p(\textrm{Data}|\theta)p(\theta)}{P(\textrm{Data})} $$ so

$$p(\theta |\textrm{Data}) \propto p(\textrm{Data}|\theta)p(\theta)$$

can we say that the posterior distribution will have also belong, say to the exponential family of distributions?

Your prior will be in the form:

$$p(\theta) = {(\frac{\phi}{2\pi}})^{0.5}\, e^{\frac{-\phi(\theta-\mu)^2}{2}} $$

which uses the parametrization of the Normal Distribution with precision $\phi$.

This, in turn, is in the same algebraic form as your Likelihood:

$$P(y|w, \phi) = (\frac{\phi}{2\pi})^{0.5} \exp ^{-0.5 e_{i}\phi e_{i}}$$

Therefore, per my answer to your first question, the posterior will be in this exponential form.

$$p(\theta |\textrm{Data}) \propto {(\frac{\phi}{2\pi}})^{0.5}\, e^{\frac{-\phi(\theta-\mu)^2}{2}} * (\frac{\phi}{2\pi})^{0.5} \exp ^{-0.5 e_{i}\phi e_{i}}$$

P.S: I realised the parameterisations in the formulae are incosistent, but got to run now. The statement on the algebraic form holds.

$\endgroup$
  • $\begingroup$ Thanks for the answer Zhubarb! May I ask why do people go for expensive methods like MCMC to characterise such a posterior? If the posterior has such a form, should it not be possible to get its moments easily? Also, is it guaranteed that such a posterior would be unimodal (considering my priors and the likelihood are modelled in this way)? $\endgroup$ – Luca Jan 2 '14 at 18:14
  • $\begingroup$ @Luca, there was a similar question that i answered here: stats.stackexchange.com/questions/79547/… $\endgroup$ – Zhubarb Jan 3 '14 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.