20
$\begingroup$

Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?

$\endgroup$
24
$\begingroup$

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

$\endgroup$
0
$\begingroup$

I don't know if I could list them all. The exponential, normal and binomial come to mind and they all fall into the class of exponential families. The exponential family has its sufficient statistic in the exponent and the mle is often a nice function of this sufficient statistic.

$\endgroup$
  • 8
    $\begingroup$ This question is incredibly broad but it appears the OP may be asking what characterizes a distribution that has a closed-form solution for the MLE rather than asking for an exhaustive list. In any case, an exhaustive list isn't even possible. $\endgroup$ – Macro Jul 11 '12 at 14:14
  • 2
    $\begingroup$ It's not always a "nice function", for example, the sufficient statistic of the beta distribution is $[\log x\; \log (1-x)]^{\rm T}$, from which numerical methods are required to find the shape parameters $a$ and $b$. $\endgroup$ – Neil G Jul 12 '12 at 1:28
  • $\begingroup$ Thnaks Neil for pointing that out. I guess not all exponential family distributions have closed form solutions. $\endgroup$ – Michael Chernick Jul 12 '12 at 1:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.