2
$\begingroup$

If $X$ is a random variable with PDF $f(x;\Theta)$, then we define the likelihood function as:

$$L(\Theta; x) = f(x;\Theta)$$

As I understand, we say that $\Theta^*$ is the true MLE estimator if $\Theta^*$ meets the following condition

$$\Theta^* = \arg\max_{\Theta} L(\Theta; x)$$

As such, MLE estimators have the following desired properties:

  • Asymptotic Normality (useful for creating confidence interval)
  • Minimum Variance (Cramer-Rao bound, i.e. the ML estimator will have the lowest possible variance out of any similar estimator)
  • Consistency (ML estimators with bigger sample sizes have higher probabilities of being closer to their real values)
  • Invariance under Transformation

From a practical perspective, we try to determine $\Theta^*$ numerically. Although there are many popular situations where the problem of determining $\Theta^*$ is a Convex Optimization problem (i.e. I heard that all likelihood functions from probability distributions belonging to exponential family are convex), I am sure there are other such problems where the problem is not necessarily convex (I am guessing in hierarchical regression models, or mixture models).

What I am getting at, is that inevitably, sometimes the approximate numerical solution $\Theta^{**}$ will not be equal to the true ML solution $\Theta^*$. In such situations, I am interested in knowing:

  • Is it still reasonable to expect $\Theta^{**}$ will be asymptotically normal, minimum variance, consistent and invariant?
  • Suppose two different people attempt to optimize the likelihood function in two different ways (e.g. two different numerical algorithms, two different parametrizations of the likelihood function, etc.) and come up with estimates $\Theta^{***}$ and $\Theta^{****}$, whereas the true ML solution is $\Theta^*$. Lets say that $|\Theta^{***} - \Theta^*| < |\Theta^{****} - \Theta^*|$. In other words, $\Theta^{***}$ is closer to the true ML solution compared to $\Theta^{***}$. If $\frac{\Theta^{***}}{\Theta^{****}} = a$, can we say that $\Theta^{***}$ is more asymptotically normal (i.e. requires smaller $n$ to be normal), more consistent (i.e. requires smaller $n$ to be closer to the true value) and lesser variance than $\Theta^{****}$ by a factor of $a$?
$\endgroup$
7
  • 3
    $\begingroup$ It might help to explore the conditions required to demonstrate the first three assertions you make at the outset, because they are not true in general. $\endgroup$
    – whuber
    Commented Jan 14 at 16:40
  • $\begingroup$ I am confused. I thought under certain regularity conditions, the first 3 conditions are usually true? $\endgroup$ Commented Jan 14 at 18:00
  • 4
    $\begingroup$ Right: and everything depends on those "certain regularity conditions." $\endgroup$
    – whuber
    Commented Jan 14 at 18:28
  • $\begingroup$ i was just thinking about this idea.... is it reasonable to expect "mle style properties" from an "approximate mle solution"? (i.e. obtained from numerical optimization algorithms) $\endgroup$ Commented Jan 14 at 18:34
  • 1
    $\begingroup$ It would depend both on the specifics of the problem and on the numerical algorithm used. $\endgroup$
    – whuber
    Commented Jan 15 at 15:00

1 Answer 1

2
+50
$\begingroup$

Indeed for some maximum likelihood estimators we have

$$\sqrt{n}(\hat\theta - \theta) \to \mathcal{N}(0,I^{-1})$$

from which those properties follow.

What I am getting at, is that inevitably, sometimes the approximate numerical solution $\Theta^{**}$ will not be equal to the true ML solution $\Theta^*$. In such situations, I am interested in knowing:

  • Is it still reasonable to expect $\Theta^{**}$ will be asymptotically normal, minimum variance, consistent and invariant?

When the MLE is computed approximately then the properties may not need to hold. However, if the error $(\Theta^{**} - \Theta^*)$ is negligible (and decreases for increasing $n$), which is not a bad assumption for an approximation method, then the properties will hold.

can we say that $\Theta^{***}$ is more asymptotically normal

For one of the properties, 'the minimum variance', one might think that it is weird that the property still holds for an approximation. Shouldn't the estimate have a larger variance? However, note that this property is about the asymptotic distribution, the distribution when $n$ approaches infinity. For a fixed value of $n$, the MLE doesn't need to be the minimum variance estimator.

These asymptotic properties will be the same when the error of the approximation decreases sufficiently fast to zero for larger $n$. Whether the error of the one approximation is larger than the other doesn't matter.

You are either asymptotical normal or not, there is no gray area, where we can be more or less asymptotical normal.

However, what can be compared instead are convergence rates and variances at finite $n$. (and again, the MLE doesn't need to have the property to have the fastes convergence rate or being the minimum variance estimator)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.