Which distributions have closed-form solutions for maximum likelihood estimation?

Question

Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?

Answer 1

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness$^\dagger$: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

--

$\dagger$ link expired. See whuber's comment.

Answer 2

I don't know if I could list them all. The exponential, normal and binomial come to mind and they all fall into the class of exponential families. The exponential family has its sufficient statistic in the exponent and the mle is often a nice function of this sufficient statistic.