26
$\begingroup$

Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum values $a$ and $b$ but of unknown parameters $\mu$ and $\sigma^2$. If $X$ followed a non-truncated distribution, the maximum likelihood estimators $\widehat\mu$ and $\widehat\sigma^2$ for $\mu$ and $\sigma^2$ from $S$ would be the sample mean $\widehat\mu = \frac{1}{N} \sum_i S_i$ and the sample variance $\widehat\sigma^2 = \frac{1}{N} \sum_i (S_i - \widehat\mu)^2$. However, for a truncated distribution, the sample variance defined in this way is bounded by $(b-a)^2$ so it is not always a consistent estimator: for $\sigma^2 > (b-a)^2$, it cannot converge in probability to $\sigma^2$ as $N$ goes to infinity. So it seems that $\widehat\mu$ and $\widehat\sigma^2$ are not the maximum-likelihood estimators of $\mu$ and $\sigma^2$ for a truncated distribution. Of course, this is to be expected since the $\mu$ and $\sigma^2$ parameters of a truncated normal distribution aren't its mean and variance.

So, what are the maximum likelihood estimators of the $\mu$ and $\sigma$ parameters of a truncated distribution of known minimum and maximum values?

$\endgroup$
  • $\begingroup$ Are you sure about your analysis? I think you are making an invalid assumption: for the truncated situation, the MLE of $\sigma^2$ is no longer the sample variance (and, in general, the MLE of $\mu$ is no longer the sample mean)! $\endgroup$ – whuber Jan 30 '13 at 16:17
  • $\begingroup$ whuber: I know, this is precisely my question: what are the MLEs of $\sigma^2$ and $\mu$ in the truncated case? Adding a sentence to insist on this. $\endgroup$ – a3nm Jan 30 '13 at 16:26
  • 1
    $\begingroup$ There isn't a closed form solution. All you can do is numerically minimize the log likelihood. But this is qualitatively no different than many other models, such as logistic regression, which also have no closed form solution. $\endgroup$ – whuber Jan 30 '13 at 16:32
  • $\begingroup$ whuber: If this is true, this is pretty disappointing. Do you have references about the lack of closed form solutions? Are there closed-form estimators that are not maximum likelihood but are at least consistent (and optionally unbiased?). $\endgroup$ – a3nm Jan 30 '13 at 16:38
  • 1
    $\begingroup$ @whuber: Can you at least simplify your samples into sufficient statistics so that the minimization is fast? $\endgroup$ – Neil G Jan 30 '13 at 16:45
27
$\begingroup$

Consider any location-scale family determined by a "standard" distribution $F$,

$$\Omega_F = \left\{F_{(\mu, \sigma)}: x \to F\left(\frac{x-\mu}{\sigma}\right) \mid \sigma \gt 0\right\}.$$

Assuming $F$ differentiable we readily find that the PDFs are $\frac{1}{\sigma}f\left((x-\mu)/\sigma\right)dx$.

Truncating these distributions to restrict their support between $a$ and $b$, $a \lt b$, means that the PDFs are replaced by

$$f_{(\mu, \sigma; a,b)}(x) = \frac{f\left(\frac{x-\mu}{\sigma}\right)dx}{\sigma C(\mu, \sigma, a, b)}, a \le x \le b$$

(and are zero for all other values of $x$) where $C(\mu, \sigma, a, b) = F_{(\mu,\sigma)}(b) - F_{(\mu,\sigma)}(a)$ is the normalizing factor needed to ensure that $f_{(\mu, \sigma; a, b)}$ integrates to unity. (Note that $C$ is identically $1$ in the absence of truncation.) The log likelihood for iid data $x_i$ therefore is

$$\Lambda(\mu, \sigma) = \sum_i \left[\log{f\left(\frac{x_i-\mu}{\sigma}\right)} - \log{\sigma}-\log{C(\mu, \sigma, a, b)}\right].$$

Critical points (including any global minima) are found where either $\sigma=0$ (a special case I will ignore here) or the gradient vanishes. Using subscripts to denote derivatives, we may formally compute the gradient and write the likelihood equations as

$$\eqalign{ 0 &= \frac{\partial\Lambda}{\partial\mu} &= \sum_i \left[\frac{-f_\mu\left(\frac{x_i-\mu}{\sigma}\right)}{f\left(\frac{x_i-\mu}{\sigma}\right)} -\frac{C_\mu(\mu,\sigma,a,b)}{C(\mu,\sigma,a,b)}\right] \\ 0 &= \frac{\partial\Lambda}{\partial\sigma} &= \sum_i \left[\frac{-f_\sigma\left(\frac{x_i-\mu}{\sigma}\right)}{\sigma^2f\left(\frac{x_i-\mu}{\sigma}\right)} -\frac{1}{\sigma}-\frac{C_\sigma(\mu,\sigma,a,b)}{C(\mu,\sigma,a,b)}\right] }$$

Because $a$ and $b$ are fixed, drop them from the notation and write $nC_\mu(\mu, \sigma, a, b)/C(\mu, \sigma,a,b)$ as $A(\mu,\sigma)$ and $nC_\sigma(\mu, \sigma, a, b)/C(\mu, \sigma,a,b)$ as $B(\mu, \sigma)$. (With no truncation, both functions would be identically zero.) Separating the terms involving the data from the rest gives

$$\eqalign{ -A(\mu,\sigma) &= \sum_i \frac{f_\mu\left(\frac{x_i-\mu}{\sigma}\right)}{f\left(\frac{x_i-\mu}{\sigma}\right)} \\ -\sigma^2 B(\mu,\sigma) - n\sigma &= \sum_i \frac{f_\sigma\left(\frac{x_i-\mu}{\sigma}\right)}{f\left(\frac{x_i-\mu}{\sigma}\right)} }$$

By comparing these to the no-truncation situation it is evident that

  • Any sufficient statistics for the original problem are sufficient for the truncated problem (because the right hand sides have not changed).

  • Our ability to find closed-form solutions relies on the tractability of $A$ and $B$. If these do not involve $\mu$ and $\sigma$ in simple ways, we cannot hope to obtain closed-form solutions in general.

For the case of a normal family, $C(\mu,\sigma,a,b)$ of course is given by the cumulative normal PDF, which is a difference of error functions: there is no chance that a closed-form solution can be obtained in general. However, there are only two sufficient statistics (the sample mean and variance will do) and the CDF is as smooth as can be, so numerical solutions will be relatively easy to obtain.

$\endgroup$
  • $\begingroup$ Thanks a lot for this very detailed answer! I'm not sure I get what $f_\mu$, $f_\sigma$ , $C_\mu$, and $C_\sigma$ are, could you define them? Also, it's obvious but to be precise maybe you could say that your expression for the pdf is for $x \in [a, b]$ (and the pdf is zero outside of that). Thanks again! $\endgroup$ – a3nm Jan 30 '13 at 17:37
  • 1
    $\begingroup$ The usual longer notation is $C_\mu = \frac{\partial}{\partial\mu}C(\mu,\sigma,a,b)$, etc: as announced, it is a derivative. I will make the second change you suggest because it's an important clarification, thanks. $\endgroup$ – whuber Jan 30 '13 at 17:38
  • $\begingroup$ Also, since your answer is more general than the one I expected, I edited my question to insist less on the case of normal distributions. Thanks again for your effort. $\endgroup$ – a3nm Jan 30 '13 at 17:43
  • 1
    $\begingroup$ It was easier to explain at this level of generality compared to focusing on the Normal distributions! Computing the derivatives and showing the precise form of the CDF are unnecessary distractions (although useful when you start actually coding the numerical solution). $\endgroup$ – whuber Jan 30 '13 at 17:45
  • 1
    $\begingroup$ Thanks for fixing! You missed one of them; could you review my edit? $\endgroup$ – a3nm Jan 31 '13 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.