1
$\begingroup$

Upon studying the ML estimator this concept still confuses me.

First define an asymptotic covariance matrix for the MLE estimator (just as an example, we have two parameters $\beta$ and $\sigma^2$, $x_i$ is a vector of observations, see the derivation at the bottom):

$$V = I(\beta, \sigma^2)^{-1} =\begin{bmatrix} \frac{\sigma^2}{x_i x_i'} & 0 \\ 0 & 2 \sigma^4 \end{bmatrix}$$

As it can be seen the variables are uncorrelated, given that the variance is per definition obtained from the matrix, and our assumption is that it's normally distributed:

$$ \sqrt N (\beta_p-\hat \beta) \rightarrow ^d N(0,\frac{\sigma^2}{x_i x_i'})$$

$$ \sqrt N (\sigma_p^2 - \hat \sigma^2) \rightarrow ^d N(0, 2 \sigma^4)$$

My confusion is that generally speaking, shouldn't asymptotic variance matrix be defined as something that happens as N grows. Thus the diagonals of the matrix should be divided by N (and perhaps even probability limit taken to obtain the true asymptotic value, which in this case would be 0 for both estimates).

Are the different definitions at odds here or am I misunderstanding something here?

$\\$

Here is the full derivation of obtaining the information matrix, since it could be a other source of confusion:

$$L(y_1... y_n|x_i; \beta, \sigma^2) = \prod_{i=1}^{n} \frac{1}{2\pi \sigma^2}e^{(-\large \frac{y_i - x_i' \beta }{2 \sigma^2})^2}$$

Take logs and then the derivative, to get the score vector:

$$ \frac{\delta (log L)}{\delta \hat \beta} = \sum_{i=1}^nx_i(\frac {y_i-x_i' \hat\beta}{\sigma^2})$$

$$ \frac{\delta (log L)}{\delta \hat \beta} = \frac {N}{2 \sigma^2} +\sum_{i=1}^nx_i(\frac {y_i-x_i' \hat\beta}{2 \sigma^4})^2$$

At this point it's somehow fine to remove the sum signs, don't know what that is based on (please inform me if you do). Finally, put the two results into a vector and multiply the vector with its inverse, then take the expectation:

$$\large \text E [\begin{bmatrix} x_i(\frac {y_i-x_i' \hat\beta}{\sigma^2}) \\ \frac {N}{2 \sigma^2} +(\frac {y_i-x_i' \hat\beta}{2 \sigma^4})^2 \end{bmatrix}\begin{bmatrix} x_i'(\frac {y_i-x_i' \hat\beta}{\sigma^2}) & \frac {N}{2 \sigma^2} +(\frac {y_i-x_i' \hat\beta}{2 \sigma^4})^2 \end{bmatrix}]$$

Calculating the result will result in the information matrix, the asymptotic covariance matrix being its inverse.

$\endgroup$
6
  • $\begingroup$ You're missing a lot of context and definitions that are needed to make sense of this question. What does "$x_i x_i^\prime$" mean? What is "$\beta$"? $\endgroup$ – whuber Apr 18 '16 at 22:20
  • $\begingroup$ Thanks for the edits. I am still a little taken aback by the presence of the superfluous subscript "$i$". And for $V$ to be the information matrix I believe you must be assuming the observations are univariate, independent, and Normally distributed: not all information matrices for two-parameter Maximum Likelihood problems look like this! Presumably "$N$" (on the left hand sides, anyway) is the count of those observations. Now, as to your question: what exactly do you think those arrows mean if they do not refer to arbitrarily large values of $N$?? $\endgroup$ – whuber Apr 18 '16 at 22:38
  • $\begingroup$ @whuber The information matrix is derived using the BHHH estimator, using the normal distribution likelihood function (equivalent to OLS). The subscript merely carries over from $x_i' \beta$, which is the regression equation. I think the arrows are referring to the shape of the distribution, not it's actual variance (which must change as N grows - that result can't be right). $\endgroup$ – Dole Apr 18 '16 at 22:55
  • 1
    $\begingroup$ It sounds like you may have some confusion over the notion of convergence in distribution. Typically in problems of interest, if you have a quantity whose mean is fixed but whose variance decreases as $\frac{1}{n}$, it will converge to a constant; this is not useful. By subtracting that mean and multiplying the difference by $\sqrt{n}$, you can get convergence to a fixed distribution. $\endgroup$ – Glen_b Apr 19 '16 at 12:55
  • 1
    $\begingroup$ Consider a simple mean, $\mu$ for iid observations with mean $\mu$ & variance $σ^2$. As $n\to\infty$, $\bar{X}_n$ converges to a point mass at $\mu$ which is not especially useful for use in calculations at large $n$, while $\sqrt{n}(\bar{X}_n-\mu)$ gives convergence to $N(0,σ^2)$. If you're interested in the variance of $\bar{X}$ at some finite $n$, then yes, you'd divide $σ^2$ back by $n$ again. There's nothing actually confusing in what's happening aside from such inconsistencies as the way people use terms like asymptotic variance, but usually it's either defined or obvious from context. $\endgroup$ – Glen_b Apr 19 '16 at 23:44
1
$\begingroup$

You are missing a lot of context here (presumably taken from linear regression) so I will focus solely on the asymptotic distribution:$^\dagger$

$$\sqrt N (\sigma^2 - \hat{\sigma}^2) \overset{\text{Dist}}{\rightarrow} \text{N}(0, 2 \sigma^4). \quad \quad$$

The left-hand-side here is a scaled version of the estimation error, which is affected by the size $N$. As $N \rightarrow \infty$ we obtain convergence in distribution to the distribution shown on the right-hand-side, which does not depend on $N$. This is sensible because although the distribution of the scaled estimation error should depend on $N$, its limit should not.

It is possible to re-frame the asymptotic result as an approximating distribution that becomes more and more accurate in the limit. (Indeed, this is the main value of an asymptotic distribution.) If $N$ is large we have:

$$\ \ \quad \sigma^2 - \hat{\sigma}^2 \overset{\text{Approx}}{\sim} \text{N} \bigg( 0, \frac{2 \sigma^4}{N} \bigg).$$

As you can see from this approximating distribution, the distribution of the estimation error is centred around zero (reflecting an unbiased estimator). As $N$ becomes larger, the (approximate) distribution of the estimation error has a lower variance, so it tends to get smaller. This gives us some useful consistency properties for the estimator, and it accords with our intuition that estimator becomes more accurate as we get more data.


$^\dagger$ Note that this notation is shorthand for the following formal mathematical statement:

$$\quad \quad \lim_{N \rightarrow \infty} \mathbb{P} \Big( \sqrt N (\sigma_p^2 - \hat \sigma^2) \leqslant \epsilon \Big) = \Phi \bigg( \frac{\epsilon}{\sqrt{2} \sigma^2} \bigg) \quad \quad \quad \text{for all } \epsilon \in \mathbb{R}.$$

where the function $\Phi$ is the cumulative distribution function for the standard normal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.