I'm curious what would be the posterior distribution having prior dstribution as a mixture of two guassian with the likelihood dist as a gaussian. In other words:

  • Likelihood: $p(x|\mu,\sigma) = \mathcal{N}(\mu,\sigma^2)$
  • Prior: $p(\mu) = a\cdot\mathcal{N}(0,1)+b\cdot\mathcal{N}(1,1)$
  • Posterior: $p(\mu|x)=???$

Also, I would like to know the MAP objective of it. For example, giving a guassian prior amounts to the $L^2$ regularization, then which kind of regularization with prior with gaussian mixture model would induce?


You can rewrite the prior as a Gaussian hyper prior: $$ p(\mu|i) = \mathrm{N}(i,1), \text{where } i \in \{0,1\} $$ and set a prior for $i$, say $$ p_0(i) = \begin{cases}a, & i=0, \\ b, & i=1. \end{cases} $$ First you could ignore the prior on $i$ and compute the posterior measure analytically parameterised in $i$. In this case, you would have just a Gaussian prior and a Gaussian likelihood, so you can compute the posterior $$ p(\mu|x,i) = \frac{p(x|\mu)p(\mu|i)}{p(x|i)} \propto p(x|\mu)p(\mu|i) $$ analytically using Bayes' formula (and the cheat sheet Wikipedia entry on conjugate priors).

Note that $p(\mu|x,i)$ is still Gaussian. To get to the actual posterior $p(\mu|x)$ we want to get rid of the $I$. This can be achieved using the following formula: $$ p(\mu|x) = \sum_{i=0}^1 p(i|x)\cdot p(\mu|x,i), \tag 1 $$ where $$ p(i|x) = \frac{p_0(i)\cdot p(x|i)}{\sum_{j=0}^1 p_0(j)\cdot p(x|j)}. $$ is the posterior measure over $i$ given the data $x$.

In any case, the posterior $p(\mu|x)$ will be again a Gaussian mixture with two components. This you can see in equation (1): We have a convex combination of two Gaussian measures there.

The question about the regularisation is somewhat more tricky. You get an $L^2$ regularisation in the Gaussian prior case, since the negative log-prior is an $L^2$-norm. Here, we have the regulariser $R(\mu)$ given by $$ R(\mu) = -\log(p(\mu)) = -\log\left(a \frac{1}{\sqrt{2\pi}} \exp(-\frac12\|\mu\|^2) + b \frac{1}{\sqrt{2\pi}} \exp(-\frac12\|\mu-1\|^2)\right). $$ I do not actually see a way to simplify this term. It is definitely something non-standard to do as a regularisation.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.