Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have recently begun to look into Bayesian Analysis, and, although I'm beginning to get to grips with the general framework (i.e. $\text{posterior} \propto \text{likelihood} \times \text{prior}$), I'm still having some difficulties in playing around with distributions.

I hope to gain a more concrete understanding of these processes through worked examples; what follows is a template-like rendering of the kinds of problems I've seen in various books and online resources - I hope it makes sense.

Assume I want to carry out a Bayesian analysis on some process. Data ($x$) gathered on this process over a certain time period suggests that the process is normally distributed with variance of $\sigma$ (known); however, the goal of the research is to discover the value of the mean ($\mu$) (unknown). Another researcher is asked his/her opinion on the same process and suggests a mean value of $\mu_{0}$ with standard deviation of $\sigma_{0}$.

  • Now, assume I wanted to specify a gamma prior distribution from the fellow researcher's opinion. What would the pdf look like in this instance? (i.e. How do I embody the mean and variance into the gamma distribution?)

  • Furthermore, for completeness, what would the likelihood and posterior pdfs of $\mu$ look like?

As you can see, it's mostly the manipulation of the distributions that are causing my difficulties.

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

Ok, I'm assuming you mean the fellow researcher's opinion is that the mean of the process is $\mu_0$ with a standard deviance of that opinion of $\sigma_0$ (rather than the researcher thinks the standard deviation of the process you are modelling is $\sigma_0$, since according to your question you know the process has standard deviation $\sigma$). In that case, you are specifying a Gamma prior for $\mu$ which should have $E[\mu]=\mu_0$ and $Var[\mu]=\sigma_0^2$.

Now, if $\mu \sim Gamma(\alpha,\beta)$, the mean and variance are given by:

\begin{equation} E[\mu] = \frac{\alpha}{\beta}, Var[\mu] = \frac{\alpha}{\beta^2} \end{equation}

So you need to set $\alpha/\beta = \mu_0$ and $\alpha/\beta^2 = \sigma_0^2$, and then solve for $\alpha$ and $\beta$ to get an appropriate choice of parameters to reflect the researcher's opinion in your prior (I think you should get $\alpha = \mu_0^2 / \sigma_0^2$ and $\beta = \mu_0 / \sigma_0^2$).

So the problem looks like this (for prior $\pi(\mu)$, likelihood $f(\mathbf{x}|\mu)$ and posterior $\pi(\mu|\mathbf{x})$, assuming $n$ independent observations):

\begin{eqnarray} \pi(\mu) &\propto& \mu^{\alpha - 1} \exp\{-\beta \mu\} \mathbb{1}_{\{\mu>0\}} \\ f(\mathbf{x}|\mu) &\propto& \exp\{-\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2\} \\ \pi(\mu|\mathbf{x}) &\propto& \mu^{\alpha - 1} \exp\{-\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 -\beta \mu \} \mathbb{1}_{\{\mu>0\}} \end{eqnarray}

The posterior given here is up to a constant of proportionality - I don't think the Gamma distribution is conjugate for the Normal mean. So to perform inference for $\mu$ you would need a sampling based approach (I would suggest rejection sampling here using a Gamma proposal density, see section 2.3 in Chapter 2 of Introducing Monte Carlo Methods with R by Robert & Casella (2009) for a good introduction to rejection sampling.)

To be clear, the conjugate prior for the mean of the Normal distribution is the Normal distribution, so if you had a prior $\mu \sim N(\mu_0,\sigma_0)$ then you would also get a Normal posterior density that you could find analytically (see any Bayesian Inference text book for the derivation and explanation, I like Peter Hoff A First Course in Bayesian Statistical Methods (2009)).

share|improve this answer
    
Great answer! It was precisely what I was looking for. Also, I'll be sure to check out the references. Thanks a million! –  user9171 Oct 11 '12 at 19:53
    
There are a few things that I was hoping to clarify; however, unfortunately, the character limit for comments is not sufficient for my queries. To work around this, I have opened a new question thread. If you would like to view, the thread can be found at: stats.stackexchange.com/questions/39238/… Thanks again for your original response! –  user9171 Oct 11 '12 at 20:10
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.