# Which distributions have closed-form solutions for maximum likelihood estimation?

Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?

-

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta)$$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

-
And for those that don't have closed forms, we could use the EM Algorithm. For example, consider the zero-inflated poisson moddel: stats.stackexchange.com/questions/32133/… –  Damien Jul 11 '12 at 21:21
It's not always a "nice function", for example, the sufficient statistic of the beta distribution is $[\log x\; \log (1-x)]^{\rm T}$, from which numerical methods are required to find the shape parameters $a$ and $b$. –  Neil G Jul 12 '12 at 1:28