Derive posterior density function with Jeffrey's prior for theta

Question

I need a guide on how to derive the posterior distribution for $\theta$ and checking whether it is proper. I have been given that the likelihood function is

$$L(\theta; x) = \theta \exp(−\theta x).$$

You decide to perform Bayesian inference on $\theta$ given a realisation $x = \{x_i\}_{i=1}^N$ of an i.i.d. sample $X_1,\ldots,X_N$ of size $N = 100.$ You do not wish to take into consideration any prior knowledge about $\theta,$ so you choose to use the Jeffrey’s prior for $\theta,$ given by $$\pi J (\theta) \propto 1/\theta.$$

If I multiply the likelihood and the prior I would only get $\exp(-\theta x)$ which does not resemble any distributions I know but I found a similar question here (Question 2b).

Which says that it is the improper $\rm Gamma(0,0)$ distribution and the posterior is $\textrm{Gamma}(n,n\bar x).$ How does it become $\rm Gamma(0,0)?$ I found a relating question which from my understanding is that it is actually not $0$ but rather a number approaching very close to $0$ thus I can write the posterior as $\textrm{Gamma}(n,n\bar x)?$

Answer 1

A gamma distribution for $\theta$ with shape $\alpha$ and rate $\beta$ has density $f(\theta)=\frac{\beta^\alpha}{\Gamma(\alpha)}\theta^{\alpha-1} \exp(-\beta \theta)$ which, by dropping multiplicative terms not involving $\theta$, we can say is proportional to $\theta^{\alpha-1} \exp(-\beta \theta)$.

If $\alpha=0$ and $\beta=0$ we might say that would be proportional to $\theta^{0-1} \exp(-0\times \theta) =\frac1\theta$ like the prior in the question, noting that this is improper. Improper priors are not much of a theoretical issue in themselves, though improper posterior densities can be (e.g a Haldane $\text{Beta}(0,0)$ prior for the probability parameter of a binomial distribution does not make much sense until you observe at least one success and one failure). If you are using numerical methods, an improper prior distribution may in some cases be impractical if you need to draw values from it; that is not the case here, since you are using a conjugate distribution, and the posterior distribution is proper even with a single observation.

If you had a single observation $x$ and an improper density for $\theta$ proportional to $\frac1\theta$ then your posterior density for $\theta$ would as you say be proportional to $\exp(-\theta x)$ which (so it integrated to $1$) actually be a density of $\pi(\theta \mid x)= x \exp(-\theta x)$ for $x \ge 0$, i.e. an exponential distribution with rate $x$. But in fact you have $n$ observations.

The likelihood of the observations is proportional to $\prod \left(\theta \exp( - \theta x_i )\right)=\theta^{n} \exp\left( - \theta \sum x_i\right)$ so the posterior density for $\theta$ is proportional to $\theta^{n-1} \exp\left( - \theta \sum x_i\right)$ and so is a gamma distribution with shape $\alpha =n$ and rate $\beta = \sum x_i = n\bar x$.