# MLE of an exponential distribution in discrete case

Consider the one parameter exponential family on a finite sample space $S$: $$p(x;\theta)=\frac{e^{\theta x}}{\sum_{x\in S} e^{\theta x}}, \theta\in \mathbb{R}.$$

My objective is to find an MLE for $\theta$ having observed a sample $x_1,\dots,x_n$ from $p(\cdot,\theta)$. In the continuous case (when $\theta$ is restricted to strictly negative and $x>0$) one can easily solve the likelihood equation and see that the MLE is negative of reciprocal of the sample mean.

When I do the same here, I obtain that the $\theta$ for which $p(x_1,\dots,x_n;\theta)$ is maximum satisfies $$\frac{\sum_x xe^{\theta x}}{\sum_x e^{\theta x}}=\frac{1}{n}\sum_i x_i.$$ Is it possible to find MLE from here in a closed form expression? LHS is mean with respect to $p(\cdot,\theta)$. Is the mean known in terms of $\theta$? Any help is appreciated.

• Without a knowledge of the S it looks like it is not possible. – mookid Mar 4 '14 at 14:57

## 1 Answer

In general there is no closed form solution. This can be seen by substituting $$u=e^\theta \hspace{2em}\text{and}\hspace{2em} z=\frac{1}{n}\sum_i x_i$$ and rearranging the equation which leads to $$\sum_x (x-z)\,u^x = 0$$ Under the assumption that the elements $x$ of the sample space are positive integers, this is a power series. A special case would be $S:=\{0,1,2\}$ where the solution can be found by solving the quadratic equation. Since it is known that equations with integral powers of order greater than four in general don't have a closed form solution also the equation in your question does not have one.

• Thank you for the answer. Can't we say anything about the MLE of $\theta$ in terms of the sample mean? (I am suspecting something because it is an exponential family). – Kumara Mar 4 '14 at 15:57
• Just because we have an exponential family doesn't mean we have closed form MLEs. Consider the logistic regression scenario (MLEs obtained using Fisher Scoring in R) – bdeonovic Mar 4 '14 at 15:58
• @Benjamin: Thanks for the reply. I am not asking for a closed form solution now. If we denote the solution of $\frac{\sum_x xe^{\theta x}}{\sum_x e^{\theta x}}=\frac{1}{n}\sum_i x_i$ by $\hat{\theta_{\text{ML}}}$, is there any connection between $\hat{\theta_{\text{ML}}}$ and $\frac{1}{n}\sum_i x_i$, like, $\hat{\theta_{\text{ML}}}/(\frac{1}{n}\sum_i x_i)\to 1$ as $n\to \infty$? – Kumara Mar 5 '14 at 9:37