6
$\begingroup$

Given the Laplace distribution parametrized by $\mu$ and $b$,

$f(x\mid \mu ,b)={\frac {1}{2b}}\exp \left(-{\frac {|x-\mu |}{b}}\right)\,\!$ ,

I know that $\hat \mu$, the maximum likelihood estimator of $\mu$, is the sample median. But how do we find the standard error of the MLE? I'm not sure how to proceed since the Fisher Information is undefined when $x = \hat \mu$, which, for my purposes, I must assume will happen.

I'm aware of the answers provided to the question here. What I'm looking for in general is a method to compute the standard errors when the log-likelihood is not twice differentiable.

$\endgroup$
  • 3
    $\begingroup$ Isn't the Fisher Information the expectation of the logarithmic derivative of the density? In this case, since the function will fail to be differentiable only on a set of measure zero, this shouldn't affect its applicability. $\endgroup$ – whuber Apr 2 '18 at 23:33
  • $\begingroup$ @whuber yes, but in practice the observed Fisher Information is often used in place of the Fisher Information (though please correct me if I am mistaken). In any case, I'd like to know how to find the standard error when the Information evaluated at the MLE does not exist. $\endgroup$ – set_seed_1234 Apr 3 '18 at 0:18
8
$\begingroup$

Appeal to the Fisher information gives you an asymptotic approximation to the standard error. As whuber correctly points out in the comments, so long as this function is almost surely differentiable, that should be sufficient to obtain your result.

However, in the present case it is also possible to obtain the exact distribution of the MLE via first principles methods, without appeal to the asymptotic theory of MLEs. Your MLE is the median, so its distribution can be obtained using standard distributional results for order statistics, where the underlying distribution is continuous. To derive the result we will assume an odd number of observations for simplicity in dealing with the median.

In this case we have $n = 2k+1$ for some non-negative integer $k$ and the MLE is $\hat{\mu} = X_{(k+1)}$. We let $f$ and $F$ be the respective density and distribution functions for the (zero-mean) sample distribution. For IID values from a continuous distribution we then have:

$$\begin{equation} \begin{aligned} \mathbb{V}( \hat{\mu}) = \mathbb{E}( (\hat{\mu} - \mu)^2 ) &= \int \limits_{-\infty}^\infty t^2 f(t) \text{Beta}(F(t) | k+1, k+1) dt \\[6pt] &= \frac{(2k+1)!}{k! k!} \int \limits_{-\infty}^\infty t^2 f(t) F(t)^k (1-F(t))^k dt \\[6pt] &= \frac{(2k+1)!}{k! k!} \int \limits_{-\infty}^\infty t^2 f(t) (F(t)-F(t)^2)^k dt \\[6pt] &= \frac{(2k+1)!}{k! k!} \sum_{i=0}^k {k \choose i} (-1)^{k-i} \int \limits_{-\infty}^\infty t^2 f(t) F(t)^i F(t)^{2k-2i} dt \\[6pt] &= \frac{(2k+1)!}{k! k!} \sum_{i=0}^k {k \choose i} (-1)^{k-i} \int \limits_{-\infty}^\infty t^2 f(t) F(t)^{2k-i} dt. \\[6pt] \end{aligned} \end{equation}$$

For the Laplace distribution we have $F(t)(1-F(t)) = \tfrac{1}{2} \exp(- |t|/b) - \tfrac{1}{4} \exp(- 2|t|/b)$ and $f(t) = \tfrac{1}{2b} \exp(- |t|/b)$ so that:

$$\begin{equation} \begin{aligned} \mathbb{V}( \hat{\mu}) &= \frac{(2k+1)!}{k! k!} \sum_{i=0}^k {k \choose i} (-1)^{k-i} \int \limits_{-\infty}^\infty t^2 f(t) F(t)^{2k-i} dt \\[6pt] &= 2 \frac{(2k+1)!}{k! k!} \sum_{i=0}^k {k \choose i} (-1)^{k-i} \int \limits_0^\infty t^2 f(t) F(t)^{2k-i} dt \\[6pt] &= b^2 \frac{(2k+1)!}{k! k!} \sum_{i=0}^k {k \choose i} (-1)^{k-i} \Big( \frac{1}{2} \Big)^{2k-i} \int \limits_0^\infty t^2 \exp (-(2k-i+1)t) dt \\[6pt] &= b^2 \frac{(2k+1)!}{2^{k-1} k! k!} \sum_{i=0}^k {k \choose i} \frac{(-1 /2)^{k-i} }{(2k-i+1)^3}. \\[6pt] \end{aligned} \end{equation}$$

This gives us a closed form (finite sum) expression for the variance of the MLE estimator (i.e., the sample median). (It is important to be careful of rounding error when evaluating the variance expression, since it involves a product of very large and very small terms.) A related expression has been investigated in some excellent analysis by Claude Liebovici in this related question, which shows that $\mathbb{V}( \hat{\mu}) \approx b^2 / n$ as $n \rightarrow \infty$. This limiting result accords with the asymptotic theory of the sample median, which has $\mathbb{V}(\hat{\mu}) \rightarrow 1 / (4n f(\mu)^2) = b^2 / n$.

$\endgroup$
  • $\begingroup$ Thank you for the thorough answer, as well as the example. The only thing I cannot follow is your calculation of the standard error of the sample mean at the very end of your post. Is that a typo? $\endgroup$ – set_seed_1234 Apr 3 '18 at 16:39
  • $\begingroup$ Yeah, that bit was wrong. Removed. $\endgroup$ – Ben - Reinstate Monica Apr 3 '18 at 22:24
  • 1
    $\begingroup$ Looking at math.stackexchange.com/questions/2721263/… I have been playing a little and it "seems" that the constant is $\frac 14$. $\endgroup$ – Claude Leibovici Apr 4 '18 at 7:06
  • $\begingroup$ @Claude: Owing to investigation of the asymptotic variance, I noticed an error in the last step of my variance expression where I had omitted an additional multiplicative term of 2 (i.e., my expression was half of what it should have been). This means that the summation expression is now twice as big as the one you investigated. Results have been adjusted accordingly. $\endgroup$ – Ben - Reinstate Monica Apr 18 '18 at 1:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.