I am taking a course in Statistical Theory of Science. We have an exercise where we are comparing "Classical" and Bayesian parameter estimation.

We have a sample $(x_1,...,x_5) = (0.28,0.30,0.94,0.42,1.76)$ from an exponential distribution with parameter $\theta$, the pdf is given as

$$f(x|\theta) =\theta e^{-\theta x}, \quad \theta, x > 0$$

We need to estimate $\theta$ by choosing an appropriate prior, combining it with the likelihood and estimating it from the posterior distribution.

My question is about the prior, what would be appropriate? My guess is that it should be a Gamma distribution based on the fact that our parameter exists in $\mathbb{R}^+$. Is this correct?

If the above is correct, am I supposed to add a guess for the parameters in the Gamma? Or should they stay as variables?

How do I interpret "combine with the likelihood", is this the calculation of the posterior?

  • $\begingroup$ Any prior is "appropriate". There are no good and bad priors! $\endgroup$ – Xi'an Dec 21 '19 at 8:54

The Gamma prior is certainly a good option as it is conjugate with the exponential likelihood. This means that the posterior will also be a Gamma distribution, and the update (prior $\rightarrow$ posterior) do not rely on any simulation.

Say you have a prior $\Gamma(a,b)$ whose density is:

$$ p(\theta ; a,b) \propto \theta^{a-1}e^{-\theta b} $$

and the exponential likelihood based on observations $(X_1,\dots,X_n)$: \begin{align*} p(X \mid \theta) &= \prod \theta e^{-\theta X_i} \\ &= \theta^n e^{-\theta \sum X_i} \end{align*}

What we are interested in is the posterior distribution of $\theta$, that is the distribution of $\theta$ after observing the data.
We get this posterior from the bayes theorem: $$ p(\theta \mid X) = \frac{p(\theta) p(X \mid \theta)}{p(X)} $$ which is sometimes written as $p(\theta \mid X) \propto p(\theta) p(X \mid \theta)$.
Here $p(\theta)$ is the prior distribution and $p(X \mid \theta)$ is the likelihood of the model.

Thus the posterior distribution of $\theta \mid X$ is proportional to the product of the prior and the likelihood (and I think this is what is meant by combine with the likelihood): \begin{align*} p(\theta \mid X) &\propto p(\theta ; a,b) p(X \mid \theta) \\ & \propto \theta^{a-1}e^{-\theta b} \theta^n e^{-\theta \sum X_i} \\ & \propto \theta^{a+n-1} e^{-\theta(b+\sum X_i)} \end{align*} The last line, seen as a function of $\theta$, is proportional to the density of a $\Gamma(a+n, b+ \sum X_i)$, which is the posterior distribution of $\theta$.

The posterior is completely determined an you can easily compute posterior mean/median or credible intervals without relying on any simulation.

You have to choose the parameter $(a,b)$ for the prior distribution of $\theta$. This should reflect you prior belief on the plausible values of $\theta$. This parameters $(a,b)$ are fixed, but can sometimes be considered themselves as random, but this is advanced bayesian statistics.

  • $\begingroup$ Thanks for this answer, very good! If I do not have any prior belief, as this is just an exercise with no context, what numbers for $a$ and $b$ would make a good choice? We are supposed to compare the result with estimation using MLE. $\endgroup$ – Jon Lachmann Dec 20 '19 at 13:38
  • $\begingroup$ To repeat myself, there is no good or bad choice of $(a,b)$. Bayesian analysis is relative to the prior measure, the posterior indicates how the data modifies this prior measure. Within Bayesian analysis it thus makes no sense to compare two priors. $\endgroup$ – Xi'an Dec 21 '19 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.