Compute conjugate prior from the sample distribution

Question

I feel like this question might be marked as duplicate because I see many similar incurring in that fate but I'll try anyway. I would say I did not find anything similar.

I have been thought a procedure to find conugate prior distributions that is based on sufficient statistics. The idea is to compute the likelihood of the sample, then identifying the sufficient statistic using Neyman factorization theorem and in the end substitute some hyperparameter for the sufficient statistic in the function $g(\theta, T(x))$, where $\theta$ is the parameter of interest and $T(X)$ is the sufficient statistic.

To give an example, I have the following exponential distribution

\begin{gather} p(y_t\mid\alpha) = \alpha\, \exp\{-\alpha y_t\}\mathbb{1}_{(0,\infty)}(y_t) \end{gather}

Then, the likelihood function is (given $y_t$ are iid)

\begin{gather} L(\alpha) = \alpha^T \exp\left\{-\alpha\sum y_t\right\} \mathbb{1}_{(0,\infty)}(\max(y_t)) \end{gather}

Using Neyman factorization theorem, we can factorize the likelihood as $g(\alpha,T(x))=\alpha^T \exp\{-\alpha\sum y_t\}$ and $c(y)=\mathbb{1}_{(0,\infty)}(\max(y_t))$ so that our sufficient statistic is $T(X) = \sum y_t$.

Then, the conjugate prior to this model should be \begin{gather} \pi(\alpha)=g(\alpha,\eta)=\alpha^T \exp\{-\alpha\eta\} \end{gather}

where $\eta$ is the hyperparameter.

I tried to compute the posterior to check if the family is the same but I got this

\begin{gather} p(\alpha\mid y_t) = \alpha^{2T} \exp\left\{-\alpha\left(\eta+\sum y_t\right) \right\} \end{gather}

which doesn't seem to be an exponential distribution to me.

Now, my question is: should I insert a random parameter $\eta$ or should it be something meaningful, maybe related to the distribution at stake? Or are there issues in my way of proceding?

Answer 1

Your prior is \begin{align} & \text{constant} \times \alpha^T \exp(-\eta\alpha)\, d\alpha & & \text{for } \alpha>0 \tag 1 \\[8pt] = {} & \frac 1 {T!\eta^{T+1}} (\eta\alpha)^T \exp(-\eta\alpha) \big( \eta\, d\alpha\big) & & \text{for } \alpha > 0. \tag 2 \end{align} This is a gamma distribution.

The likelihood as you have given it is $$ L(\alpha) = \alpha^T \exp\left( -\alpha\sum_{t=1}^T y_t \right). $$ But notice that $\eta$ is not the only hyperparameter in the prior; in effect $T$ is also a hyperparameter. Just why you have chosen it to be equal to the sample size is not clear, probably because there is no good reason. Suppose we re-write $(1)$ as $$ \text{constant} \times \alpha^S \exp(-\eta\alpha) \, d\alpha \quad \text{for }\alpha>0. $$ It is best to use the form $(1)$ rather than $(2)$ at this point and find the normalizing constant only after finding the posterior distribution except for that constant.

Now multiply and you get a gamma distribution whose "shape" parameter is $S+T.$

Both the prior and the posterior are gamma distributions.