1
$\begingroup$

I'm reading the book Information Theory, Inference and Learning Algorithms. In Section 22.1, the author gives an example of finding the MLE of the mean of an univariate Gaussian, and then obtaining the error bar of it, given the data and the standard variation.

The related text is:

If we Taylor-expand the log likelihood about the maximum, we can define approximate error bars on the maximum likelihood parameter: we use a quadratic approximation to estimate how far from the maximum-likelihood parameter setting we can go before the likelihood falls by some standard factor, for example $e^{1/2}$ , or $e^{4/2}$. In the special case of a likelihood that is a Gaussian function of the parameters, the quadratic approximation is exact.

Then comes Example 22.2:

Find the second derivative of the log likelihood with respect to $\mu$, and find the error bars on $\mu$, given the data and $\sigma$.

The solution to this example in the text is:

Comparing this curvature with the curvature of the log of a Gaussian distribution over $\mu$ of standard deviation $\sigma_{\mu}$, $\exp(-\mu^2/(2\sigma_{\mu}^2))$, which is $-1/\sigma_{\mu}^2$, we can deduce that the error bars on $\mu$ (derived from the likelihood function) are $$\sigma_{\mu} = \frac{\sigma}{\sqrt{N}}$$

I don't understand the above procedure of finding the error bars by "comparing the curvature", what's the principle behind it?

$\endgroup$
0
$\begingroup$

Without being entirely sure, it seems to me that he compares the standard deviation according to the Gaussian curvature with the result of the second derivative, mentioned a bit before in the text. In order for the two terms to be equal, the result must be the one you specify. Solving with respect to $\sigma_{\mu}^2$, you can indeed verify this for yourself. Hope this helps!

$\endgroup$
0
$\begingroup$

I couldn't follow his solution method but here is another way of getting the same result: Let $ L(\mu) $ be the log likelihood as a function of $\mu$. (See equation 22.4 in the David MacKay book .) Further let $\mu_{m}$ be the MLE, which is also the sample mean (See equation 22.6) of the N iid samples from $ \mathcal{N} (\mu,\sigma)$ where $\sigma$ is known. Then, by definition of MLE, in the quadratic-approximation based on the Taylor expansion of $L(\mu)$ around $\mu_m$, we would have $L'(\mu_m) = 0$, so that : $$ L(\mu) = L(\mu_m) + L''_n(\mu_m)*(\mu-\mu_m)^2/2 = L(\mu_m) - N*(\mu-\mu_m)^2/(2\sigma^2) $$ since $L''(\mu) = -N/\sigma^2$

This means $$ L(\mu) = L(\mu_m) - ((\mu - \mu_m)/(\sigma/\sqrt{N}))^2)/2 $$

(Note that $ (\mu - \mu_m)/(\sigma/\sqrt{N}) \sim \mathcal{N}(0,1)$ since $\mu_m$ is a sample mean, and by CLT.)

Now, the likelihood at $\mu = \mu_m + \sigma/\sqrt{N}$ is $e^{L(\mu)} = e^{L(\mu_m)}/e^{1/2} $, where $e^{L(\mu_m)}$ is the maximum likelihood, which is the desired result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.