Specifying the Form of Prior, Likelihood and Posterior Distributions for Bayesian Analysis

Question

I have recently begun to look into Bayesian Analysis, and, although I'm beginning to get to grips with the general framework (i.e. $\text{posterior} \propto \text{likelihood} \times \text{prior}$), I'm still having some difficulties in playing around with distributions.

I hope to gain a more concrete understanding of these processes through worked examples; what follows is a template-like rendering of the kinds of problems I've seen in various books and online resources - I hope it makes sense.

Assume I want to carry out a Bayesian analysis on some process. Data ($x$) gathered on this process over a certain time period suggests that the process is normally distributed with variance of $\sigma$ (known); however, the goal of the research is to discover the value of the mean ($\mu$) (unknown). Another researcher is asked his/her opinion on the same process and suggests a mean value of $\mu_{0}$ with standard deviation of $\sigma_{0}$.

• Now, assume I wanted to specify a gamma prior distribution from the fellow researcher's opinion. What would the pdf look like in this instance? (i.e. How do I embody the mean and variance into the gamma distribution?)

• Furthermore, for completeness, what would the likelihood and posterior pdfs of $\mu$ look like?

As you can see, it's mostly the manipulation of the distributions that are causing my difficulties.

Answer 1

Ok, I'm assuming you mean the fellow researcher's opinion is that the mean of the process is $\mu_0$ with a standard deviance of that opinion of $\sigma_0$ (rather than the researcher thinks the standard deviation of the process you are modelling is $\sigma_0$, since according to your question you know the process has standard deviation $\sigma$). In that case, you are specifying a Gamma prior for $\mu$ which should have $E[\mu]=\mu_0$ and $Var[\mu]=\sigma_0^2$.

Now, if $\mu \sim Gamma(\alpha,\beta)$, the mean and variance are given by:

\begin{equation} E[\mu] = \frac{\alpha}{\beta}, Var[\mu] = \frac{\alpha}{\beta^2} \end{equation}

So you need to set $\alpha/\beta = \mu_0$ and $\alpha/\beta^2 = \sigma_0^2$, and then solve for $\alpha$ and $\beta$ to get an appropriate choice of parameters to reflect the researcher's opinion in your prior (I think you should get $\alpha = \mu_0^2 / \sigma_0^2$ and $\beta = \mu_0 / \sigma_0^2$).

So the problem looks like this (for prior $\pi(\mu)$, likelihood $f(\mathbf{x}|\mu)$ and posterior $\pi(\mu|\mathbf{x})$, assuming $n$ independent observations):

\begin{eqnarray} \pi(\mu) &\propto& \mu^{\alpha - 1} \exp\{-\beta \mu\} \mathbb{1}_{\{\mu>0\}} \\ f(\mathbf{x}|\mu) &\propto& \exp\{-\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2\} \\ \pi(\mu|\mathbf{x}) &\propto& \mu^{\alpha - 1} \exp\{-\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 -\beta \mu \} \mathbb{1}_{\{\mu>0\}} \end{eqnarray}

The posterior given here is up to a constant of proportionality - I don't think the Gamma distribution is conjugate for the Normal mean. So to perform inference for $\mu$ you would need a sampling based approach (I would suggest rejection sampling here using a Gamma proposal density, see section 2.3 in Chapter 2 of Introducing Monte Carlo Methods with R by Robert & Casella (2009) for a good introduction to rejection sampling.)

To be clear, the conjugate prior for the mean of the Normal distribution is the Normal distribution, so if you had a prior $\mu \sim N(\mu_0,\sigma_0)$ then you would also get a Normal posterior density that you could find analytically (see any Bayesian Inference text book for the derivation and explanation, I like Peter Hoff A First Course in Bayesian Statistical Methods (2009)).