Definition of asymptotic variance

Question

Upon studying the ML estimator this concept still confuses me.

First define an asymptotic covariance matrix for the MLE estimator (just as an example, we have two parameters $\beta$ and $\sigma^2$, $x_i$ is a vector of observations, see the derivation at the bottom):

$$V = I(\beta, \sigma^2)^{-1} =\begin{bmatrix} \frac{\sigma^2}{x_i x_i'} & 0 \\ 0 & 2 \sigma^4 \end{bmatrix}$$

As it can be seen the variables are uncorrelated, given that the variance is per definition obtained from the matrix, and our assumption is that it's normally distributed:

$$ \sqrt N (\beta_p-\hat \beta) \rightarrow ^d N(0,\frac{\sigma^2}{x_i x_i'})$$

$$ \sqrt N (\sigma_p^2 - \hat \sigma^2) \rightarrow ^d N(0, 2 \sigma^4)$$

My confusion is that generally speaking, shouldn't asymptotic variance matrix be defined as something that happens as N grows. Thus the diagonals of the matrix should be divided by N (and perhaps even probability limit taken to obtain the true asymptotic value, which in this case would be 0 for both estimates).

Are the different definitions at odds here or am I misunderstanding something here?

$\\$

Here is the full derivation of obtaining the information matrix, since it could be a other source of confusion:

$$L(y_1... y_n|x_i; \beta, \sigma^2) = \prod_{i=1}^{n} \frac{1}{2\pi \sigma^2}e^{(-\large \frac{y_i - x_i' \beta }{2 \sigma^2})^2}$$

Take logs and then the derivative, to get the score vector:

$$ \frac{\delta (log L)}{\delta \hat \beta} = \sum_{i=1}^nx_i(\frac {y_i-x_i' \hat\beta}{\sigma^2})$$

$$ \frac{\delta (log L)}{\delta \hat \beta} = \frac {N}{2 \sigma^2} +\sum_{i=1}^nx_i(\frac {y_i-x_i' \hat\beta}{2 \sigma^4})^2$$

At this point it's somehow fine to remove the sum signs, don't know what that is based on (please inform me if you do). Finally, put the two results into a vector and multiply the vector with its inverse, then take the expectation:

$$\large \text E [\begin{bmatrix} x_i(\frac {y_i-x_i' \hat\beta}{\sigma^2}) \\ \frac {N}{2 \sigma^2} +(\frac {y_i-x_i' \hat\beta}{2 \sigma^4})^2 \end{bmatrix}\begin{bmatrix} x_i'(\frac {y_i-x_i' \hat\beta}{\sigma^2}) & \frac {N}{2 \sigma^2} +(\frac {y_i-x_i' \hat\beta}{2 \sigma^4})^2 \end{bmatrix}]$$

Calculating the result will result in the information matrix, the asymptotic covariance matrix being its inverse.

Answer 1

You are missing a lot of context here (presumably taken from linear regression) so I will focus solely on the asymptotic distribution:$^\dagger$

$$\sqrt N (\sigma^2 - \hat{\sigma}^2) \overset{\text{Dist}}{\rightarrow} \text{N}(0, 2 \sigma^4). \quad \quad$$

The left-hand-side here is a scaled version of the estimation error, which is affected by the size $N$. As $N \rightarrow \infty$ we obtain convergence in distribution to the distribution shown on the right-hand-side, which does not depend on $N$. This is sensible because although the distribution of the scaled estimation error should depend on $N$, its limit should not.

It is possible to re-frame the asymptotic result as an approximating distribution that becomes more and more accurate in the limit. (Indeed, this is the main value of an asymptotic distribution.) If $N$ is large we have:

$$\ \ \quad \sigma^2 - \hat{\sigma}^2 \overset{\text{Approx}}{\sim} \text{N} \bigg( 0, \frac{2 \sigma^4}{N} \bigg).$$

As you can see from this approximating distribution, the distribution of the estimation error is centred around zero (reflecting an unbiased estimator). As $N$ becomes larger, the (approximate) distribution of the estimation error has a lower variance, so it tends to get smaller. This gives us some useful consistency properties for the estimator, and it accords with our intuition that estimator becomes more accurate as we get more data.

---

$^\dagger$ Note that this notation is shorthand for the following formal mathematical statement:

$$\quad \quad \lim_{N \rightarrow \infty} \mathbb{P} \Big( \sqrt N (\sigma_p^2 - \hat \sigma^2) \leqslant \epsilon \Big) = \Phi \bigg( \frac{\epsilon}{\sqrt{2} \sigma^2} \bigg) \quad \quad \quad \text{for all } \epsilon \in \mathbb{R}.$$

where the function $\Phi$ is the cumulative distribution function for the standard normal distribution.