2
$\begingroup$

I know that if it exists, a regular, unbiased estimator $T$ for $\tau(\theta)$ attains the Cramér-Rao Lower Bound (next, CRLB) if and only if I can decompose the score function as follows: $S(\theta)=\frac{\partial}{\partial\theta}logf_\mathbf{x}(\mathbf{x};\theta)=k(\theta,n)[T(\mathbf{X})-\tau(\theta)]$, where $k(\theta;n)$ is a generic function.

In particular, what is the link between CRLB and this last property and MLEs ${\hat{\theta}}$? I mean, is it possible that is something like that ${\hat{\theta}}$ always satisfies the decomposition above and thus it always reaches the CRLB?

$\endgroup$
2
$\begingroup$

It's difficult to identify the correct level of rigor for an answer. I added the "regularity" condition to your question, since there are unbiased estimators that beat the Cramer-Rao bound.

Regular exponential families have score functions for parameters that take this linear form. So we have some idea that this notation is not arbitrary; it comes from estimating "usual" things that produce reasonable outcomes.

As you know, obtaining maxima of a functional (like a likelihood or log-likelihood) involves finding the root of its derivative if it's smooth and the root is continuous. For regular exponential families, the linear form means the solution is obtainable in closed form.

When the score has that form, its expectation is 0 and its variance is an information matrix. It was a revelation to me to think of a score as a random variable, but indeed it's a function of $X$. Using the Cauchy–Schwartz inequality, you can show that any biased estimator is the sum of an unbiased estimator and the bias of the original estimator. Therefore the variance is greater in the sum of these two functions.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ ok that's clear. Now, if you can read my comment to the guy above, I meant that talking about MLEs $\endgroup$ – PhDing Dec 21 '15 at 18:56
2
$\begingroup$

In short, the decomposition comes from an application of the Cauchy-Schwartz inequality. Indeed, if you look back at the proof of the CRLB you will see that it is used to derive the bound. But then we know that the inequality becomes an equality if one of the vectors is a multiple of the other and this is precisely why the score function is decomposed like that. I will leave the specifics to you but this is the central idea.

I know that proofs are often avoided in the first reading but this one is actually easy to follow and not to mention quite instructive. So by all means take a look.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, I know where the bound comes from. My question is more specific: once you know the decomposition of the score function, being an MLE means anything? $\endgroup$ – PhDing Dec 21 '15 at 18:47
  • $\begingroup$ @Alessandro What do you mean "means anything"? The mle does not always satisfy the condition so the CRLB might not be attainable.. $\endgroup$ – JohnK Dec 21 '15 at 18:50
  • $\begingroup$ I am citing from my notes "MLEs in general satisfies $S(\theta)=0$, so since $K(\theta,n)$ is a non-zero constant, it must be true that $T(\mathbf{X})=\tau(\hat\theta)$" $\endgroup$ – PhDing Dec 21 '15 at 18:54
  • $\begingroup$ @Alessandro what about that is unclear to you? $S(\theta) = 0$ for the ML estimate, $K$ non-zero, $T(X) - \tau(\hat{\theta})$ must be 0. $\endgroup$ – AdamO Dec 21 '15 at 19:00
  • 1
    $\begingroup$ @Alessandro only if it's unbiased. The ML estimate of the variance of a sequence of IID $\mathcal{N}(\mu, \sigma^2)$ RVs is a good example of biased ML estimators. They're asymptotically efficient, but it's not the UMVUE. Of course, you'd find the score equations do not satisfy the usual form. $\endgroup$ – AdamO Dec 22 '15 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.