22
$\begingroup$

Which distributions have closed-form solutions for the maximum likelihood estimates of the parameters from a sample of independent observations?

$\endgroup$

2 Answers 2

27
$\begingroup$

Without any appreciable loss of generality we may assume that the probability density (or mass) $f(x_i)$ for any observation $x_i$ (out of $n$ observations) is strictly positive, enabling us to write it as an exponential

$$ f(x_i) = \exp{(g(x_i,\theta))}$$

for a parameter vector $\theta = (\theta_j)$.

Equating the gradient of the log likelihood function to zero (which finds stationary points of the likelihood, among which will be all interior global maxima if one exists) gives a set of equations of the form

$$\sum_i\frac{d g(x_i, \theta)}{d\theta_j} = 0,$$

one for each $j$. For any one of these to have a ready solution, we would like to be able to separate the $x_i$ terms from the $\theta$ terms. (Everything flows from this key idea, motivated by the Principle of Mathematical Laziness$^\dagger$: do as little work as possible; think ahead before computing; tackle easy versions of hard problems first.) The most general way to do this is for the equations to take the form

$$\sum_i \left(\eta_j(\theta) \tau_j(x_i) - \alpha_j(\theta)\right) = \eta_j(\theta)\sum_i \tau_j(x_i) - n \alpha_j(\theta) $$

for known functions $\eta_j$, $\tau_j$, and $\alpha_j$, for then the solution is obtained by solving the simultaneous equations

$$\frac{n\alpha_j(\theta)}{\eta_j(\theta)}= \sum_i \tau_j(x_i)$$

for $\theta$. In general these will be difficult to solve, but provided the set of values of $\left(\frac{n\alpha_j(\theta)}{\eta_j(\theta)}\right)$ give full information about $\theta$, we could simply use this vector in place of $\theta$ itself (thereby somewhat generalizing the idea of a "closed form" solution, but in a highly productive way). In such a case, integrating with respect to $\theta_j$ yields

$$g(x, \theta) = \tau_j(x)\int^\theta \eta_j(\theta) d\theta_j - \int^\theta \alpha_j(\theta) d\theta_j + B(x, \theta_j')$$

(where $\theta_j'$ stands for all the components of $\theta$ except $\theta_j$). Because the left hand side is functionally independent of $\theta_j$, we must have that $\tau_j(x)=T(x)$ for some fixed function $T$; that $B$ must not depend on $\theta$ at all; and the $\eta_j$ are derivatives of some function $H(\theta)$ and the $\alpha_j$ are derivatives of some other function $A(\theta)$, both of them functionally independent of the data. Whence

$$g(x, \theta) = H(\theta)T(x) - A(\theta) + B(x).$$

Densities that can be written in this form make up the well-known Koopman-Pitman-Darmois, or exponential, family. It comprises important parametric families, both continuous and discrete, including Gamma, Normal, Chi-squared, Poisson, Multinomial, and many others.

--

$\dagger$ link expired. See whuber's comment.

$\endgroup$
4
  • $\begingroup$ And for those that don't have closed forms, we could use the EM Algorithm. For example, consider the zero-inflated poisson moddel: stats.stackexchange.com/questions/32133/… $\endgroup$
    – Damien
    Commented Jul 11, 2012 at 21:21
  • 1
    $\begingroup$ Whuber, could you please check the link (and provide an alternative,if you deem so) to Principle of Mathematical Laziness, for it seems the link has expired and apparently even the Archive has no backup? $\endgroup$ Commented Apr 15 at 13:24
  • 1
    $\begingroup$ @User Thank you; that link is long gone, but is unnecessary: the PML is explained here. $\endgroup$
    – whuber
    Commented Apr 15 at 13:55
  • $\begingroup$ Thanks @whuber for clarification. $\endgroup$ Commented Apr 15 at 14:05
0
$\begingroup$

I don't know if I could list them all. The exponential, normal and binomial come to mind and they all fall into the class of exponential families. The exponential family has its sufficient statistic in the exponent and the mle is often a nice function of this sufficient statistic.

$\endgroup$
3
  • 9
    $\begingroup$ This question is incredibly broad but it appears the OP may be asking what characterizes a distribution that has a closed-form solution for the MLE rather than asking for an exhaustive list. In any case, an exhaustive list isn't even possible. $\endgroup$
    – Macro
    Commented Jul 11, 2012 at 14:14
  • 4
    $\begingroup$ It's not always a "nice function", for example, the sufficient statistic of the beta distribution is $[\log x\; \log (1-x)]^{\rm T}$, from which numerical methods are required to find the shape parameters $a$ and $b$. $\endgroup$
    – Neil G
    Commented Jul 12, 2012 at 1:28
  • 1
    $\begingroup$ Thnaks Neil for pointing that out. I guess not all exponential family distributions have closed form solutions. $\endgroup$ Commented Jul 12, 2012 at 1:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.