Suppose $x\mid\theta \sim \operatorname{Gamma}(\frac{n}{2},2\theta)$ and $\theta \sim$ inverse Gamma$(\alpha, \beta)$

with loss function $L(\theta, d)=\frac{(\theta-d)^2}{\theta^2}$

We wish to find the Bayes estimator, ie the estimator minimizing posterior expected loss. If it was square error loss, then it would simply be the posterior mean. But since not, we must minimize the estimator with respect to the expected posterior loss.

Issues occur in that there is no obvious distribution for the posterior or way to evaluate the integrals.

The question appeared on a past exam. User Xi'an left the very interesting note that we could transfer the denominator of the loss function to our prior and then we would have the standard square error loss. In that case it would be a matter of finding the mean of the resulting posterior distribution. But I am seeing slightly different answers, as well as the answer I was given was the quotient of two non standard integrals and involved differentiating under the integral sign.

  • $\begingroup$ You should add the self-study tag to your question. Adding $\theta^{-2}$ to the quadratic loss is equivalent to adding $\theta^{-2}$ to the prior: switch to an Inverse Gamma with parameter $\alpha+2$ instead and use the posterior mean. $\endgroup$ – Xi'an Nov 2 '18 at 5:07
  • $\begingroup$ @Xi'an Tag added. Not Homework by the way. I see. Because it will all be the same when you to integrate. But is the resulting kernel a standard distribution? $\endgroup$ – Quality Nov 2 '18 at 5:11
  • $\begingroup$ Since Inverse Gamma is conjugate to Gamma in this case, everything remains standard. $\endgroup$ – Xi'an Nov 2 '18 at 5:14
  • $\begingroup$ Very nice. Suppose you didnt switch the prior, would you still get the same result, just with the ugly integrals? $\endgroup$ – Quality Nov 2 '18 at 5:23
  • $\begingroup$ This duality is covered in my book, The Bayesian Choice, Chapter 3 $\endgroup$ – Xi'an Nov 2 '18 at 11:57

All of this can be solved analytically, since your prior distribution is the conjugate prior. From the specified distributions you have:

$$L_x(\theta) = \theta^{-n/2} \exp \Big( -\frac{x}{2\theta} \Big) \quad \quad \quad \pi(\theta) \propto \theta^{-\alpha-1} \exp \Big( -\frac{\beta}{\theta} \Big).$$

So the posterior is:

$$\begin{equation} \begin{aligned} \pi(\theta|x) &\propto \theta^{-n/2} \exp \Big( -\frac{x}{2\theta} \Big) \cdot \theta^{-\alpha-1} \exp \Big( -\frac{\beta}{\theta} \Big) \\[6pt] &= \theta^{-(\alpha+n/2)-1} \exp \Big( - \Big( \beta+\frac{x}{2} \Big) \frac{1}{\theta} \Big) \\[6pt] &\propto \text{Inv-Gamma} \Big( \theta \Big| \alpha+\frac{n}{2}, \beta+\frac{x}{2} \Big). \end{aligned} \end{equation}$$

So you have $\theta|x \sim\text{Inv-Gamma}(\alpha+n/2, \beta+x/2)$, which is of the same family as your prior (i.e., this is a conjugate prior). Observation of the data updates the posterior by adding to the initial hyperparameters in the prior. We have $\theta^{-1}|x \sim\text{Gamma}(\alpha+n/2, \beta+x/2)$ which means you get the risk function:

$$\begin{equation} \begin{aligned} R(d) &\equiv \mathbb{E}( L(d(x), \theta) |x) \\[6pt] &= \mathbb{E} \Big( \frac{(\theta-d(x))^2}{\theta^2} \Big| x \Big) \\[6pt] &= \mathbb{E} \Big( 1 - \frac{2d(x)}{\theta} + \frac{d(x)^2}{\theta^2} \Big| x \Big) \\[6pt] &= 1 - 2 \mathbb{E}( \theta^{-1} | x) \cdot d(x) + \mathbb{E}( \theta^{-2} |x) \cdot d(x)^2 \\[6pt] &= 1 - 2 \cdot \frac{\alpha+\tfrac{n}{2}}{\beta+\tfrac{x}{2}} \cdot d(x) + \frac{(\alpha+\tfrac{n}{2})(\alpha+\tfrac{n}{2}+1)}{(\beta+\tfrac{x}{2})^2} \cdot d(x)^2. \\[6pt] \end{aligned} \end{equation}$$

This function is a convex quadratic equation in $d$, which can easily be minimised with respect to $d$ to give you the Bayes' estimator. Minimisation of the Bayes risk gives the estimator:

$$\hat{\theta} \equiv d(x) = \frac{\beta+\tfrac{x}{2}}{\alpha+\tfrac{n}{2}+1}.$$

  • $\begingroup$ How do you get that likilihood for n from gamma distribution? Just took a quick look $\endgroup$ – Quality Nov 2 '18 at 3:32
  • $\begingroup$ You are missing the data $x$ from your posterior $\endgroup$ – probabilityislogic Nov 2 '18 at 4:32
  • $\begingroup$ @probabilityislogic I think he just replaced x with n? $\endgroup$ – Quality Nov 2 '18 at 4:38
  • 1
    $\begingroup$ @Ben . Yes he used that convention to be the opposite. I agree it is misleading. $\endgroup$ – Quality Nov 2 '18 at 22:44
  • 1
    $\begingroup$ @Quality: Well, I think it is a fair criticism. $\endgroup$ – Ben Nov 2 '18 at 23:15

Ben's answer addresses your question completely. But I wanted to explain what happens in general. The loss function here is an example of a weighted squared error loss function. In general, a weighted squared error loss function can be written as $$L(\theta, d) = \dfrac{(\theta-d)^2}{w(\theta)}\,. $$ For example, in your situation, the weight is $w(\theta) = \theta^2$. As you mentioned, the Bayes estimator is the estimator that minimizes the expected posterior loss. That is $$d_{bayes} = \arg\min_d \int L(\theta, d) \pi(\theta|x)d\theta\,.$$

The expected posterior loss is, \begin{align*} \int L(\theta, d) \pi(\theta|x)d\theta & = \int \dfrac{(\theta - d)^2 }{w(\theta)} \dfrac{\pi(x | \theta) \pi(\theta)}{\pi(x)} d\theta\\ & = \int (\theta - d)^2 \dfrac{\pi(x | \theta)}{\pi(x)} \underbrace{\dfrac{\pi(\theta)}{w(\theta)}}_{\text{New prior kernel } \propto \pi'(\theta)}d\theta\\ & \propto \int (\theta - d)^2 \dfrac{\pi(x | \theta)}{\pi(x)} \pi'(\theta)d\theta\\ & = \int(\theta - d)^2 \pi'(\theta|x)d\theta\,. \end{align*} where $\pi'(\theta|x)$ is the new posterior distribution resulting from the new prior $\pi'(\theta)$. Then the minimizer of this new expected posterior loss, is the posterior mean with respect to $\pi'(\theta|x)$.

As probabilityislogic pointed out, the new prior may be improper, in which case the posterior may be improper, but the integral will still accurately represent the expected posterior loss.

  • $\begingroup$ There are obviously caveats here. That new prior may be improper for some choices of weight function. $\endgroup$ – probabilityislogic Nov 2 '18 at 11:09
  • $\begingroup$ @probabilityislogic Ah yes, that's correct. I'll edit the answer to indicate that. $\endgroup$ – Greenparker Nov 2 '18 at 11:47
  • $\begingroup$ It does not really matter that the prior gets proper or improper as long as the posterior loss can be minimised. $\endgroup$ – Xi'an Nov 2 '18 at 11:56
  • 1
    $\begingroup$ @Greenparker: Great answer (+1). Please check my edit to see if it accurately captures your work - reverse if needed. $\endgroup$ – Ben Nov 2 '18 at 22:35
  • 1
    $\begingroup$ Thanks @Greenparker I appreciate this answer as well. I think my issue was mainly from the abuse of notation (and simply, wrong notation). But this helps to better my understanding for the future! $\endgroup$ – Quality Nov 3 '18 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.