12
$\begingroup$

Premise: this may be a stupid question. I only know the statements about MLE asymptotic properties, but I never studied the proofs. If I did, maybe I woulnd't be asking these questions, or I maybe I would realize these questions don't make sense...so please go easy on me :)

I've often seen statements which say that the MLE estimator of a model's parameters is asymptotically normal and efficient. The statement is usually written as

$\hat{\theta}\xrightarrow[]{d}\mathcal{N}(\theta_0,\mathbf{I}(\theta_0)^{-1})$ as $N\to\infty$

where $N$ is the number of samples, $\mathbf{I}$ is Fisher information and $\theta_0$ is the parameter (vector) true value. Now, since there is reference to a true model, does this mean that the result will not hold if the model is not true?

Example: suppose I model power output from a wind turbine $P$ as a function of wind speed $V$ plus additive Gaussian noise

$P=\beta_0+\beta_1V+\beta_2V^2+\epsilon$

I know the model is wrong, for at least two reasons: 1) $P$ is really proportional to the third power of $V$ and 2) the error is not additive, because I neglected other predictors which are not uncorrelated with wind speed (I also know that $\beta_0$ should be 0 because at 0 wind speed no power is generated, but that's not relevant here). Now, suppose I have a infinite database of power and wind speed data from my wind turbine. I can draw as many samples i want, of whatever size. Suppose I draw 1000 samples, each of size 100, and compute $\hat{\boldsymbol{\beta}}_{100}$, the MLE estimate of $\boldsymbol{\beta}=(\beta_0,\beta_1,\beta_2)$ (which under my model would just be the OLS estimate). I thus have 1000 samples from the distribution of $\hat{\boldsymbol{\beta}}_{100}$. I can repeat the exercise with $N=500,1000,1500,\dots$. As $N\to\infty$, should the distribution of $\hat{\boldsymbol{\beta}}_{N}$ tend to be asymptotically normal, with the stated mean and variance? Or does the fact that model is incorrect invalidate this result?

The reason I'm asking is that rarely (if ever) model are "true" in applications. If the asymptotic properties of MLE are lost when the model is not true, then it might make sense to use different estimation principles, which while less powerful in a setting where the model is correct, may perform better than MLE in other cases.

EDIT: it was noted in the comments that the notion of true model can be problematic. I had the following definition in mind: given a family of models $f_{\boldsymbol{\theta}}(x)$ indicized by the parameter vector $\boldsymbol{\theta}$, for each model in the family you can always write

$Y=f_{\boldsymbol{\theta}}(X)+\epsilon$

by simply defining $\epsilon$ as $Y-f_{\boldsymbol{\theta}}(X)$. However, in general the error won't be orthogonal to $X$, have mean 0, and it won't necessarily have the distribution assumed in the derivation of the model. If there exists a value $\boldsymbol{\theta_0}$ such that $\epsilon$ has these two properties, as well as the assumed distribution, I would say the model is true. I think this is directly related to saying that $f_{\boldsymbol{\theta_0}}(X)=E[Y|X]$, because the error term in the decomposition

$Y=E[Y|X]+\epsilon$

has the two properties mentioned above.

$\endgroup$
  • 3
    $\begingroup$ MLE estimation is often asymptotically normal even if the model is not true, it might be consistent for the "least false" parameter values, for instance. But in such cases it wil be difficult to show efficency or other optimality properties. $\endgroup$ – kjetil b halvorsen Jan 3 '17 at 13:52
  • 1
    $\begingroup$ Before efficiency we should look at consistency. In a scenario when truth is not in your search space we need a different definition of consistency such that: d(P*, P), where d is a divergence P* is the closest model in terms of d, and P is truth. When d is KL divergence (what MLE is minimizing) for example it is known that Bayesian procedures are inconsistent (can't reach the closest model) unless model is convex. Therefore I would assume that MLE will be inconsistent as well. Therefore efficiency becomes ill defined. homepage.tudelft.nl/19j49/benelearn/papers/Paper_Grunwald.pdf $\endgroup$ – Cagdas Ozgenc Jan 3 '17 at 13:55
  • 1
    $\begingroup$ @Cagdas Ozgenc: In many cases (such as logistic regression) MLE is still consistent for the "least false" parameters. Do you have a reference for your claim about inconsistency in the nonconvex case? Would be very interested? (Likelihood function of logistic regression is convex) $\endgroup$ – kjetil b halvorsen Jan 3 '17 at 14:00
  • $\begingroup$ @kjetilbhalvorsen homepages.cwi.nl/~pdg/ftp/inconsistency.pdf It is way over my head, but it is what I understand. If my understanding is false please correct me. I am just a hobbyist after all. $\endgroup$ – Cagdas Ozgenc Jan 3 '17 at 14:10
  • 4
    $\begingroup$ I think we get in trouble when we use terms like "model is true" or "least false". When dealing with models in practice they are all approximate. If we make certain assumptions we can use mathematics to show statistical properties. There is always a conflict here between the mathematics of probability and practical data analysis. $\endgroup$ – Michael R. Chernick Jan 3 '17 at 15:10
4
$\begingroup$

I don't believe there is a single answer to this question.

When we consider possible distributional misspecification while applying maximum likelihood estimation, we get what is called the "Quasi-Maximum Likelihood" estimator (QMLE). In certain cases the QMLE is both consistent and asymptotically normal.

What it loses with certainty is asymptotic efficiency. This is because the asymptotic variance of $\sqrt n (\hat \theta - \theta)$ (this is the quantity that has an asymptotic distribution, not just $\hat \theta$) is, in all cases,

$$\text{Avar}[\sqrt n (\hat \theta - \theta)] = \text{plim}\Big( [\hat H]^{-1}[\hat S \hat S^T][\hat H]^{-1}\Big) \tag{1}$$

where $H$ is the Hessian matrix of the log-likelihood and $S$ is the gradient, and the hat indicates sample estimates.

Now, if we have correct specification, we get, first, that

$$\text{Avar}[\sqrt n (\hat \theta - \theta)] = (\mathbb E[H_0])^{-1}\mathbb E[S_0S_0^T](\mathbb E[H_0])^{-1} \tag{2}$$

where the "$0$" subscript denotes evaluation at the true parameters (and note that the middle term is the definition of Fisher Information), and second, that the "information matrix equality" holds and states that $-\mathbb E[H_0] = \mathbb E[S_0S_0^T]$, which means that the asymptotic variance will finally be

$$\text{Avar}[\sqrt n (\hat \theta - \theta)] = -(\mathbb E[H_0])^{-1} \tag{3}$$

which is the inverse of the Fisher information.

But if we have misspecification, expression $(1)$ does not lead to expression $(2)$ (because the first and second derivatives in $(1)$ have been derived based on the wrong likelihood). This in turn implies that the information matrix inequality does not hold, that we do not end up in expression $(3)$, and that the (Q)MLE does not attain full asymptotic efficiency.

$\endgroup$
  • $\begingroup$ $\text{Avar}$ is the asymptotic variance of the random variable, and $\text{plim}$ stands for convergence in probability, right? Your answer seems very interesting, but I don't understand what $\theta$ is in your context. I was referring to a case where the right value of $\theta$ simply does not exist: see my wind turbine example, where whatever the value of $\boldsymbol{\beta}=(\beta_0,\beta_1,\beta_2)$, there is no value that makes the model correct, because there's no $\beta_3$ term, and because other predictors correlated with $V$ are missing. What would $\theta$ mean in this context? $\endgroup$ – DeltaIV Jan 4 '17 at 10:48
  • $\begingroup$ sorry, the first edition of my comment was incomprehensible: now my point should be clear. In other words, if there is no "true" $\theta$, what should we intepret as $\theta$ in the expression $\sqrt n (\hat \theta - \theta)$? $\endgroup$ – DeltaIV Jan 4 '17 at 10:52
  • 1
    $\begingroup$ @DeltaIV Zero. Will the QMLE "catch" this? It depends on whetehr it will be consistent or not -and again, there is no single answer to that question $\endgroup$ – Alecos Papadopoulos Jan 4 '17 at 12:24
  • 1
    $\begingroup$ I understood. So the QMLE (if consistent) should converge to $\theta=0$: I would have thought it would converge to some "least false" parameter value, as suggested by @kjetilbhalvorsen. Can you suggest any reference on the QMLE and the equations you wrote? Thanks $\endgroup$ – DeltaIV Jan 4 '17 at 16:11
  • 1
    $\begingroup$ @DeltaIV I would suggest the exposition in Hayashi ch. 7 about Extremum Estimators, as regards MLE consistency, normality etc. As regards QMLE the topic is rather broad. For example, under "QMLE" we may indeed have also situations where we acknowledge from the start that the parameters we are estimating may not have a clear connection to any "true parameters" (but the exercise is still valid as an approximation)., and so obtain a "least false" vector as suggested. $\endgroup$ – Alecos Papadopoulos Jan 4 '17 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.