# Compute conjugate prior from the sample distribution

I feel like this question might be marked as duplicate because I see many similar incurring in that fate but I'll try anyway. I would say I did not find anything similar.

I have been thought a procedure to find conugate prior distributions that is based on sufficient statistics. The idea is to compute the likelihood of the sample, then identifying the sufficient statistic using Neyman factorization theorem and in the end substitute some hyperparameter for the sufficient statistic in the function $$g(\theta, T(x))$$, where $$\theta$$ is the parameter of interest and $$T(X)$$ is the sufficient statistic.

To give an example, I have the following exponential distribution

$$\begin{gather} p(y_t\mid\alpha) = \alpha\, \exp\{-\alpha y_t\}\mathbb{1}_{(0,\infty)}(y_t) \end{gather}$$

Then, the likelihood function is (given $$y_t$$ are iid)

$$\begin{gather} L(\alpha) = \alpha^T \exp\left\{-\alpha\sum y_t\right\} \mathbb{1}_{(0,\infty)}(\max(y_t)) \end{gather}$$

Using Neyman factorization theorem, we can factorize the likelihood as $$g(\alpha,T(x))=\alpha^T \exp\{-\alpha\sum y_t\}$$ and $$c(y)=\mathbb{1}_{(0,\infty)}(\max(y_t))$$ so that our sufficient statistic is $$T(X) = \sum y_t$$.

Then, the conjugate prior to this model should be $$\begin{gather} \pi(\alpha)=g(\alpha,\eta)=\alpha^T \exp\{-\alpha\eta\} \end{gather}$$

where $$\eta$$ is the hyperparameter.

I tried to compute the posterior to check if the family is the same but I got this

$$\begin{gather} p(\alpha\mid y_t) = \alpha^{2T} \exp\left\{-\alpha\left(\eta+\sum y_t\right) \right\} \end{gather}$$

which doesn't seem to be an exponential distribution to me.

Now, my question is: should I insert a random parameter $$\eta$$ or should it be something meaningful, maybe related to the distribution at stake? Or are there issues in my way of proceding?

• The likelihood should be a function of $\alpha,$ not of $y_t.$ Thus: $$L(\alpha) = \cdots$$ – Michael Hardy Apr 13 '19 at 17:44
• @MichaelHardy You mean I should fix the parenthesis or there is something wrong in the computations? – PhDing Apr 13 '19 at 17:48
• Maybe writing it as $L(\alpha \mid y_1,\ldots,y_t)$ would be good. It should be clear that the likleihood function is $\alpha\mapsto L(\cdots)$ rather than $y_t\mapsto L(\cdots)$ or something like that. $\qquad$ – Michael Hardy Apr 13 '19 at 18:00

## 1 Answer

Your prior is \begin{align} & \text{constant} \times \alpha^T \exp(-\eta\alpha)\, d\alpha & & \text{for } \alpha>0 \tag 1 \\[8pt] = {} & \frac 1 {T!\eta^{T+1}} (\eta\alpha)^T \exp(-\eta\alpha) \big( \eta\, d\alpha\big) & & \text{for } \alpha > 0. \tag 2 \end{align} This is a gamma distribution.

The likelihood as you have given it is $$L(\alpha) = \alpha^T \exp\left( -\alpha\sum_{t=1}^T y_t \right).$$ But notice that $$\eta$$ is not the only hyperparameter in the prior; in effect $$T$$ is also a hyperparameter. Just why you have chosen it to be equal to the sample size is not clear, probably because there is no good reason. Suppose we re-write $$(1)$$ as $$\text{constant} \times \alpha^S \exp(-\eta\alpha) \, d\alpha \quad \text{for }\alpha>0.$$ It is best to use the form $$(1)$$ rather than $$(2)$$ at this point and find the normalizing constant only after finding the posterior distribution except for that constant.

Now multiply and you get a gamma distribution whose "shape" parameter is $$S+T.$$

Both the prior and the posterior are gamma distributions.

• I am not sure I got this part "But notice that η is not the only hyperparameter in the prior; in effect T is also a hyperparameter. Just why you have chosen it to be equal to the sample size is not clear, probably because there is no good reason". If you follow the steps, T (sample size) appears because I computed the prior the way I did. – PhDing Apr 13 '19 at 22:32
• @PhDing : It looks to me as if the exponent $T$ appeared in the likelihood function in your posting, and the reason for its appearance there is made clear in your posting. However, the prior is another matter. Why the superscript $T$ is in the conjugate prior is what was not clear. – Michael Hardy Apr 14 '19 at 2:21