Consider the one parameter exponential family on a finite sample space $S$: $$p(x;\theta)=\frac{e^{\theta x}}{\sum_{x\in S} e^{\theta x}}, \theta\in \mathbb{R}.$$

My objective is to find an MLE for $\theta$ having observed a sample $x_1,\dots,x_n$ from $p(\cdot,\theta)$. In the continuous case (when $\theta$ is restricted to strictly negative and $x>0$) one can easily solve the likelihood equation and see that the MLE is negative of reciprocal of the sample mean.

When I do the same here, I obtain that the $\theta$ for which $p(x_1,\dots,x_n;\theta)$ is maximum satisfies $$\frac{\sum_x xe^{\theta x}}{\sum_x e^{\theta x}}=\frac{1}{n}\sum_i x_i.$$ Is it possible to find MLE from here in a closed form expression? LHS is mean with respect to $p(\cdot,\theta)$. Is the mean known in terms of $\theta$? Any help is appreciated.

  • $\begingroup$ Without a knowledge of the S it looks like it is not possible. $\endgroup$ – mookid Mar 4 '14 at 14:57

In general there is no closed form solution. This can be seen by substituting $$u=e^\theta \hspace{2em}\text{and}\hspace{2em} z=\frac{1}{n}\sum_i x_i$$ and rearranging the equation which leads to $$\sum_x (x-z)\,u^x = 0$$ Under the assumption that the elements $x$ of the sample space are positive integers, this is a power series. A special case would be $S:=\{0,1,2\}$ where the solution can be found by solving the quadratic equation. Since it is known that equations with integral powers of order greater than four in general don't have a closed form solution also the equation in your question does not have one.

  • $\begingroup$ Thank you for the answer. Can't we say anything about the MLE of $\theta$ in terms of the sample mean? (I am suspecting something because it is an exponential family). $\endgroup$ – Kumara Mar 4 '14 at 15:57
  • $\begingroup$ Just because we have an exponential family doesn't mean we have closed form MLEs. Consider the logistic regression scenario (MLEs obtained using Fisher Scoring in R) $\endgroup$ – bdeonovic Mar 4 '14 at 15:58
  • $\begingroup$ @Benjamin: Thanks for the reply. I am not asking for a closed form solution now. If we denote the solution of $\frac{\sum_x xe^{\theta x}}{\sum_x e^{\theta x}}=\frac{1}{n}\sum_i x_i$ by $\hat{\theta_{\text{ML}}}$, is there any connection between $\hat{\theta_{\text{ML}}}$ and $\frac{1}{n}\sum_i x_i$, like, $\hat{\theta_{\text{ML}}}/(\frac{1}{n}\sum_i x_i)\to 1$ as $n\to \infty$? $\endgroup$ – Kumara Mar 5 '14 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.